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FIVE PROBLEMS basic to the development of precise readability 
fonnulas were investigated. Cloze tests were used to determine the 
comprehension difficulties of 20 passages and of each word, inde- 
pendent clause, and sentence within each passage. A large num- 
ber of entirely new linguistic variables were derived along with 
some previously studied variables and some refined versions of 
previously studied variables. The results were as follows: First, 
nonlinear correlation techniques will have to be used in some 
readability formulas. At the word level of analysis, F tests of 
linearity showed most regressions were curvilinear. The results 
were inconclusive at the other levels of analysis. Second, readabil- 
ity formulas can predict difficulty as well for subjects at one level 
of ability as for subjects at the other levels. Third, readability 
formulas can make usefully valid predictions of the difficulties of 
individual words, independent clauses, and sentences. Fourth, the 
validity of readability formulas based entirely on linguistic varia- 
bles can be greatly improved. A multiple correlation of .934 was 
found between the linguistic variables and passage difficulty. 
Fifth, the greatest improvement in readability prediction will re- 
sult from developing more sophisticated linguistic variables. Sev- 
eral new variables exhibited correlations exceeding .y with com- 
prehension difficulty. Two correlations reached .8. 



La facilite de lire: Une nouvelle approche 

CINQ PROBLi;MES fondamentaux du developpemeni de formules 
precises pour ^valuer la facility de lire ffirent examines. Des 
6preuves “Cloze” furent employee pour determiner les difficultes 
de comprehension de 20 passages et de chaque mot, de membres 
de phrases ind^pendentes, et des phrases dans chaque passage. 
Un grand nombre de variables linguistiques enti^rement nouvelles 
furent deduites avec quelques variables pr6c6dement 6tudi6es et 
quelques versions raffin^es de variables etadi^es pr6c6deinent. Les 
r^sultats furent les suivants: premi^rement des techniques de 
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correlation non-lin^aire doivent etre employees dans quelques 
formules de la facilite de lire. Au niveau du mot de I’analyse les 
tests F de linearite montrant que le plus des rebroussement etaint 
cur\ilineaires. Deuxiemement, les formules de facilite de lire 
peuvent predire des diificultes tout aussi bien pour des sujets £l 
un certain niveau de capacite que pour des sujets a d’autres 
niveaux. Troisiemement, les formules de facUite de lire peuvent 
donner des predictions utilement valides des difiicultes de mots 
individuels, de membres de phrases independentes et de phrases. 
Quatriemement, la validite de formules de facilite de lire basee 
entidrement sur des variables linguistiques peut etre ameiiore 
enormement. Une correlation multiple de .934 fut trouvee entre 
les variables linguistiques et les difficultes de certains passages. 
Cinquiemement, le plus grande pcrfectionnement pour la pre- 
diction de la facilite d’etre lu resultera d’un developpement de 
variables linguistiques plus sophistiquees. Plusieurs nouvelles 
variables montrent des correlations plus grandes que .7 avec la 
difficulte de consprendre. Deux correlations montrent jusqu’a .8. 

Legihilidad: Un nuevo adelanto 

sE iNVEsTiGARON cinco problemas fundamentales sobre el desa- 
rrollo de fdrmulas precisas de legibilidad. Se utilizaron las pruebas 
Cloze para determinar las dificultades en comprensidn de 20 
pasajes y de cada palabra, clausula independiente, y oracidn den- 
tro de cada pasaje. Se obtuvo un gran numero de variables lin- 
guisticas completamente nuevas ademds de otras variables an- 
teriormente estudiadas. Los resultados fueron los siguientes; 
Primero, en algunas formulas de legibilidad se tendran que usar 
tecnicas de correiacion no lineal. En el analisis de letras, las 
pruebas F de linealidad mustraron que la mayoria de las re- 
gresiones eran curvilineas. Los resultados en los demas niveles 
del analisis no fueron concluyentes. Segundo, las fdrmulas de 
legibilidad pueden pronosticar dificultades lo mismo para sujetos 
en un nivel de habilidad que para sujetos en otros niveles. Ter- 
cero, las fdrmulas de ligibilidad pueden ser muy utiles para pro- 
nosticar las dificultades con palabras individuales, clausulas inde- 
pendientes y oraciones. Cuarto, la validez de fdrmulas de 
legibilidad que se basan enteramente en las variables linguisticas 
pueden mejorarsc considerablemente. Se encontrd una correiacion 
multiple de .934 entre las variables linguisticas y las dificultades 
en los pasajes. Quinto, la mejora mds notable en el prondstico de 
legibilidad resultard del desarrollo de variables linguisticas mas 
sofisticadas. Varias de las nuevas variables exhibieron corre- 
laciones que pasaban de .7 con dificultad de comprensidn. Dos de 
las correlaciones llegaron a .8. 
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One of the great challenges to scientists of this generation is 
to learn how to predict and control the difficulty of language. It is 
almost trite to say that further improvement of public and private 
life depends upon the ability to transmit ever increasing amounts of 
knowledge to an increasingly large proportion of the population. But, 
unfortunately, many adults and children fail to understand what they 
read, not because the concepts are too difficult or because they lack 
basic reading skills, but simply because of the complexity of the lan- 
guage in which those concepts are presented. The money wasted an- 
nually on materials of this sort mounts into millions of dollars. More 
appalling is the waste in human terms as, for example, when a child 
fails to learn and drops out of school, when citizens are unable to 
inform themselves on important matters of government, and when 
a worker is unemployed because he cannot keep up with advancing 
technology. But the cost of lives blighted by poverty, ignorance, crime, 
frustration, and loss of self respect is not easily calculated. Much of 
this waste must be directly attributed to the present inability to predict 
and control language difficulty. 

The ultimate objective of this and other similar research is to 
develop accurate formulas for predicting and controlling the readabil- 
ity of language. But before this objective can be attained, certain basic- 
problems must be solved. This study takes up some of these basic 
problems. The first is to determine whether various features of writing 
style are linearly related to the comprehension difficulty of language. 
The second is to find out if the strengths of these relationships change 
as a function of the reading abilities of subjects. The third is to find 
out if useful readability prediction can be made on language units as 
small as a word or an independent clause. The fourth is to gain some 
idea of how much accuracy can be attained in passage readability 
formulas when modern testing and linguistic analysis techniques are 
used to construct these formulas. The fifth is to try out several new 
types of linguistic variables (measures of features of writing style) to 
see if they are useful in predicting language difficulty. 



Background 

Quality of present readability formulas 

It is problematic whether presently available readability for- 
mulas help more than they hinder. Because these formulas are easy 
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and inexpensive to apply, they enjoy widespread use by publishers and 
educators. Publishers use them for “adjusting” the difficulty of instruc- 
tional materials, and educators use them to decide if instructional ma- 
terials are suitable for students of a given level of reading ability. Chall 
(I958) has made a strong case that the formulas are not sufficiently 
accurate to warrant either of these uses. Their validity correlations 
range from .5 to only .7, and experiments have shown that they have 
little, if any, validity when they are used as style guides for “adjusting” 
the difficulty of materials. Hence, the publishers’ “adjustments” of the 
materials probably do not have the desired effect on the actual diffi- 
culty of the materials. But the practice does mislead educators. Since 
educators use essentially »/ie same formulas as the publishers, they 
believe that the materials are suitable for their students when, in fact, 
they are not. 

Advances in research techniques 

For many years it seemed impossible to improve the formulas 
because the available research techniques were not equal to the task. 
In recent years linguists have been making rapid strides in developing 
objective ways of dealing with language. Also, the cloze test has solved 
the problem of reliably measuring language difficulty. These new tech- 
niques have made it possible to reopen this issue and to deal effec- 
tively with many of the basic problems which were beyond the reach 
of earlier techniques. 

The most serious problem encountered in earlier research was 
the measurement of comprehension difficulty of passages. The best 
practice available was to give subjects multiple choice tests over the 
passages. Lorge (I939) criticized this practice because it was uncer- 
tain whether the difficulty of the language in the test questions or the 
difficulty of the language in the passage itself was being measured. 
The matter was confounded further by the fact that it is notoriously 
easy to vary the difficulties of these tests simply by changing the al- 
ternatives to the question. 

The cloze test procedure first conceived by Taylor ( I953) was 
the crucial factor in revitalizing this research. It made possible the 
accurate measurement of the comprehension difficulty of passages. 

A cloze test is made over a passage by replacing every fifth 
word with an underlined blank of a standard length. Subjects are told 
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to write in the words they think were deleted and responses are scored 
correct when they exactly match the words deleted. A large amount of 
research has gone into the development of the cloze procedure. Since 
Rankin (I965) has written an excellent analysis of this research, only 
the most relevant studies will be cited. Fletcher (I955) showed that 
cloze tests are valid measures of comprehension ability. Bormuth 
(I962) confirmed Fletcher’s reports and indicated that cloze tests arc 
valid and highly reliable measures of the comprehension difficulty of 
passages. MacGiniti>; (I96I) found that cloze items are statistically 
independent when surrounded by four words of context. Bormuth 
(I965) found that scoring synonyms correct does not increase the 
validities of the scores. In every case these results were obtained by 
administrating the cloze tests untimed and without the subjects first 
reading the passages from which the tests were made. 

% 

Problems studied 

If the present study has a single central focus, it is to dem- 
onstrate that, by marshaling modern techniques of psychological 
measurement, linguistics, statistics, and automatic data processing, it 
is once again possible to make large and useful advances in the study 
of readability. A range of problems whose solutions remained beyond 
the reach of earlier research methods has been brought under attack 
in the present study. These problems were selected: first, because they 
are basic to achieving accurate readability prediction; and, second, 
because they represent directions in which research must proceed if 
the applications of readability are to be raised to the status of a sci- 
ence. These problems by no means represent the only directions in 
which research must proceed. For example, efforts must be made to 
discover the basic dimensions in which prose style varies. But the ef- 
fective study of such problems must await the solution of the prob- 
lems included in the present study. 

Linearity of regressions 

Precise readability predictions depend on more than just a 
knowledge of the sizes of correlations between linguistic variables and 
comprehension difficulty. Knowledge of whether or not these correla- 
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tions are linear is required. If they are curvilinear, the shapes of the 
curves must be determined. Previous investigators have uniformly be- 
haved as if the correlations were linear by using the Pearson product 
moment correlation model and by using multiple linear regression 
equations for readability formulas. 

While these investigators may have carefully inspected the 
scatter plots of their correlations for linearity, such an inspection does 
not necessarily reveal an underlying curvilinearity. These investiga- 
tors could not measure the difficulties of individual words and clauses. 
Consequently, they were forced to plot passage difficulty against lin- 
guistic variables obtained by averaging values across an entire passage. 
These averages tend to have distributions that are approximately nor- 
mal, as can be deduced from the central limits theorem; and two 
normally distributed variables usually have a linear relationship, if a 
relationship at all exists between them. While the use of such averages 
may not violate the correlation models used, it results in the loss of 
much information that could have been retained by weighting the 
variable before averaging. 

There is evidence that these underlying relationships are not 
linear. For example, King-E!lison and .Jenkins (I954) found a hy- 
perbolic relationship between the Thorndike frequencies of words and 
the recognition times of subjects. Therefore, there is reason to believe 
that the precision of readability prediction can be increased by inves- 
tigating the shapes of these relationships. 

Variable strength as a function of reading ability 

One of the most troublesome questions in readability predic- 
tion is : Do the linguistic variables that influence difficulty for persons 
at one level of reading achievement have an equal influence on diffi- 
culty for students at other levels of achievement? At least three kinds 
of speculation are possible on this issue. One theory holds that linguis- 
tic variables influence difficulty only for persons who have not yet 
achieved a high degree of skill in reading. If this were true, readability 
formulas would be applicable only to persons having low reading 
achievement. A second theory holds that one set of variables has a 
strong influence on difficulty for persons at one level of reading abil- 
ity and other sets for persons at other levels. If this theory is correct, 
a number of readability formulas must be found and each may be 
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used only for persons within a limited range of reading ability. A 
third theory states that the linguistic variables influencing difficulty for 
persons at one level of ability have an c^ual influence on difficulty for 
persons at other levels of ability. This theory requires the use of only 
one formula. Should either of the first two theories be proven correct, 
the consequences for both psycholinguistic theory and readability pre- 
diction would be far reaching. 

The literature on readability contains statements to the ef- 
fect that special readability formulas must be developed for use with 
subjects at different levels of reading ability (Smith & Dechant, 196I ). 
Chall (I958) cited two types of evidence supporting this notion. First, 
Chall pointed out that the Lorge formula seemed to be better suited 
for predicting difficulty for young children than the Dale-Chall for- 
mula, while the Dale-Chall formula is superior for older children. 
Second, Chall pointed out that Gray (I935) observed that the sizes 
of the correlations between linguistic and difficulty variables differed 
depending on the reading ability of the groups of subjects used. While 
these effects could also have been attributed to uncontrolled variables 
in fhe studies involved, the evidence was sufficiently strong to warrant 
further investigation of this question. 

Readability of small language units 

There is an acute need for readability formulas which meas- 
ure the readabilities of individual words and sentences. Such formulas 
can be applied extensively to materials such as tiles, indexes, captions, 
and test items. They may also be used to locate difficult spots within 
larger texts. However, most readability formulas have been designed 
only for measuring the readabilities of passages. The reason is not dif- 
ficult to find. Until the advent of the cloze test there was no practical 
way to measure the comprehension difficulties of individual words and 
sentences. While one formula (Forbes & Cottle, I953) purportedly 
measures the readability of test items, the manner in which it was 
constructed makes its value dubious. In the Forbes and Cottle study 
several readability formulas were applied to each item in a set of test 
items and these readabilities were averaged for each of the items. The 
average was then used as the dependent variable. The question investi- 
gated in the present study was whether it is possible to obtain useful 
readability formulas for these smaller language units. 
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Validities of readability formulas 

Probably the most important question raised in the current 
study is whether it is possible to improve present readability formulas. 
It is true that formulas currently in use were developed using relatively 
crude techniques and that they have relatively low validities. At the 
same time, many of the early investigators made expert use of the 
techniques available to them. There may be an upper limit on the 
accuracy with which difficulty can be predicted using just the objec- 
tively measurable features of language. If such a limit exists, has it 
already been reached? 

New linguistic variables 

Without question the most important advances in readabil- 
ity research should result from the development of new linguistic 
variables. Linguistic scientists have done much in recent years to im- 
prove basic understanding of language and to develop objective ways 
of describing it. Their techniques have been adapted to the study of 
psycholinguistic problems and to problems of computer translation of 
language, resulting in the development of several new and potentially 
useful variables. Several such variables were investigated. The varia- 
bles included in the present study by no means exhaust the possible 
variables that could have been included. The ones included were se- 
lected because they are representative of types of variables that could 
be developed further, if the findings suggest the effort would be worth- 
while. These variables and their abbreviations are given below. To 
conserve space, abbreviations are used in all references to the varia- 
bles in tables. 

Word depth (WOiR DEP) The method of deriving word depth was 
developed by Yngve (I960; I962). It begins with an analysis and 
diagraming of the syntactic or immediate constituent structure of a 
sentence. Then a set of counting rules is applied to the diagram of 
the structure. Because of the complexity of deriving the variable and 
because Yngve has defined it, only a general description will be of- 
fered here. 

Figure 1 shows an example of a sentence which has been 
analyzed to derive the depths of the words in it. The line diagram 
shows the syntactic relationships among the words and phrases in this 
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sentence. Note that the horizontal lines at the highest level connect the 
most distantly related elements while the lines at the lower levels con- 
nect the more immediately related constituents. 

The counting rules applied to such diagrams are based on 
several psychological assumptions about the way in which a sentence 
is produced. First, it is assumed that a sentence is produced starting at 
the top of this diagram and working downward through the structure 
until the first word is produced. The remaining grammatical struc- 
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The dark brown bear sniffed hungrily. 
(2) (3) (2) ( 1 ) ( 1 ) (o) 

Total depth, 9; net depth, 3; mean depth, 1.5. 

FIG, 1, Illustration of the Word Depth Analysis, 



tures and words are then produQcd from left to right. At each level in 
this structure there are two elements that must be produced. For exam- 
ple, at the top level both a subject and a predicate must be produced 
if the sentence is to be grammatically acceptable. Only one of these 
structures can be expanded and produced at a time and this must be 
in a left to right” order. Consequently, if the sentence is to be com- 
pleted correctly, the speaker must remember that the production of 
a predicate must follow the production of the subject. Similarly at each 
lower level in the structure, he must remember to complete any struc- 
ture which he has begun. The depth of a given word is found by count- 
ing the number of structures a speaker has started but not completed 
at the time that he is producing that word. For example, in Figure 1 
the depth of the word The is 2 because he must remember that two 
structures must follow, a noun and a predicate. But the word dark 
has a depth of three because he must remember to produce an adjec- 
tive, a noun, and a predicate to complete the sentence correctly. Thus, 
it is assumed in this model that the difficulty of the structure of a sen- 
tence derives from the number of grammatical facts that must be 
stored in the memory as the sentence is produced or interpreted. 
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The word depth measure can be used in several ways. The 
depths of the words can be used individually, or they can be totaled 
across the sentence. In the latter case, total word depth is necessarily 
related to measures of sentence length, but the necessity of this rela- 
tionship can be eliminated by subtracting from total word depth the 
number of words in the sentence. This variable was labeled net word 
depth (WOR DEP— WOR). The total can also be divided by the num- 
ber of words in a sentence or a passage to obtain the mean word depth 
(WOR DEP/WOR). 

The word depth variable has been used in only one other 
readability study (Bormuth, I964). A correlation of .78 was found 
between the mean word depths and the comprehension difficulties of 
a set of passages. When the vocabulary difficulties and mean sentence 
lengths of the passages were experimentally held constant, the corre- 
lation was .77, indicating that the effect of mean word depth is inde- 
pendent of sentence length. 

Letter redundancy (LET RED) Carterette and Jones (I963) de- 
scribed a measure of redundancy of passages and prepared a computer 
program for performing rather lengthy calculations involved in deriv- 
ing this measure. Again, because of the complexity of deriving this 
variable only its general description can be given here. A detailed 
description of the mathematical procedure is given by Carterette and 
Jones. 

The Letter Redundancy variable is a measure of the sequen- 
tial dependencies between pairs of letters in the words of a passage. 
Suppose that the letter m occurs a number of times in a passage. The 
frequencies with which this letter is followed by each of the other let- 
ters is tabulated and transformed into probabilities, and these prob- 
abilities are then used to calculate the redundancy of the passage using 
the Shannon-Weiner (Shannon, I943) function. Such a measure of 
redundancy was calculated for pairs of letters taken four at a time 
(LET RED,j) and taken eleven at a time (LET REDn)- The purpose 
in measuring redundancy across these different numbers of letters 
was to separate the within word redundancy from the between word 
redundancy. The former was intended as a type of \/ord difficulty 
measure, while the latter was intended as a type of syntactic re- 
dundancy measure. 
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Carterette and Jones (I963) calculated LET REDn for sam- 
ples of text drawn from reading textbooks used at the first, second, 
third, and fifth gi'ade levels. The measured redundancies of the pas- 
sages correlated perfectly with the grade levels at which the books 
were used. 

Independent clause frequency (IND CLAU FRE) The IND CLAU 
FFE was derived from data gathered by Strickland (I962) who re- 
corded samples of the language of elementary school children enrolled 
in grades one through six. Strickland analyzed the grammatical pat- 
tern of each independent clause, classified it according to its type, and 
then tabulated the frequency with which each clause pattern occurred. 

The independent clauses in the passages used in the present 
study were classified using Strickland’s scheme. Each independent 
clause was then assigned a number corresponding to the frequency 
with which that pattern had been observed in Strickland’s language 
sample. The frequencies for the samples of language taken from chil- 
dren enrolled in grades one, three, and six were combined and used to 
obtain this measure of frequency. 

Figure 2 shows an example of a sentence pattern derived by 
this analysis. 



Sentence 

We found and caught the brown bear when we heard him crashing 
( 1 ) ( 2 ) (4) (Mg) 

about in the underbrush. 

(Mg continued) 

The pattern of this sentence is 1, 2, 4, Mg. 

FIG. 2. Example of Independent Clause Pattern Analysis. 



The major constituents of the sentence were classified and a 
symbol assigned to each constituent. The pattern of an independent 
clause is the ordered sequence of symbols associated with it. It should 
be noted that subordinate clauses and constituents of other major por- 
tions of clauses are left unanalyzed. Hence, this analysis accounts for 
only the gross aspects of the patterns of independent clauses. This 
variable was derived individually for each independent clause, and a 
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mean was obtained for each passage (IND CLAU FRE/IND CLAU). 

Ruddell (I963) wrote three pairs of passages which were 
carefully matched for Dale-Chall readability, subject matter content, 
and writing style, but which differed in that one member of each pair 
was written with high frequency clause patterns and the other with 
low frequency patterns. The data seemed to show that the passages 
written in low frequency patterns were more difficult to understand. 
Unfortunately, there is no way to estimate the confidence that can be 
placed in this result, since no account was taken of item sample error 
(Bormuth, I965) in the analysis. 

Letter counts (LET) Flesch (I948) found that a count of the 
mean number of syllables per word had a correlation of .66 with 
comprehension difficulty. He theorized that this count measures the 
‘abstractness” of words. A less esoteric theorist might speculate that 
the length of a word is the factor affecting its difficulty and that count- 
ing syllables in words afPords a crude measure of this factor. If length 
is the crucial factor, a more discriminating measure could be obtained 
by counting the number of letters in words. This variable can be used 
to measure the lengths of words, independent clauses, or sentences. 

Parts of speech Boder (I940) found that authors vary consid- 
erably with respect to the ratio of adjectives to verbs used in their 
writing. Carroll (I960), extending Boder’s work, found that authors 
also vary with respect to the frequency with which they use a number 
of other parts of speech. He factor analyzed a number of these varia- 
bles and found that part of speech variables seemed to measure a 
broad range of characteristics of style. However, no effort seems to 
have been made to determine whether these variables are in any way 
related to the comprehension difficulty of language. 

There was some problem in deciding what classification 
scheme should be used to classify words in the present study. F^jst, 
there was the problem of whether to use the structural or the semantic 
properties of words for classifying them. Because of their superior ob- 
jectivity, the structural properties were chosen as the principal attri- 
butes used in classifying. Then the question of what categories would 
be used arose. Almost any part of speech category can be broken down 
into a number of sub-categories; the total number of possible sub- 
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categories is very large. Using a large number of categories results in 
a more complete understanding of the influence of parts of speech on 
comprehension, but it also produces categories in which words seldom 
occur. An attenuation of variance among passages results. Therefore, 
two different sets of categories were used. One set was similar to the 
eight categories traditionally used. The other included the four form 
classes defined by Fries (I952) and a fifth group containing all other 
structural words. Thus a small number of categories was used in each 
set and some means of assessing the effects of subclasses was pro- 
vided. Within each set, the categories were put together in all possible 
ratios to each other. 

Table 1 shows the sub-categories used in classifying the words 
and the major categories into which they were combined. 



Table 1 Sub-categories and combined categories of the part of 
speech analysis 





Traditional parts of speech 


Form 


Sub^categories 


Name 


Abbreviation 


classes 


Noun 


Noun 


n 


1 


Personal Pronoun 


Pronoun 


pn 


1 


Interrogative Pronoun 


Pronoun 


pn 


s 


Demonstrative Pronoun 


Pronoun 


pn 


s 


Indefinite Pronoun 


Pronoun 


pn 


1 


Conjunctive Pronoun 


Pronoun 


pn 


s 


Verb 


Verb 


V 


2 


Auxiliary Verb 


Verb 


V 


5 


‘‘Do” Verb 


Verb 


V 


s 


Negative 


Adverb 


adv 


s 


Simple Adverb 


Adverb 


adv 


4 


Number Adverb 


Adverb 


adv 


s 


Degree Adverb 


Adverb 


adv 


s 


Introducer (there) 


Adverb 


adv 


s 


Preposition 


Preposition 


prep 


s 


Conjunction 


Conjunction 


conj 


s 


Descriptive Adjective 


Adjective 


adj 


3 


Ordinal Adjective 


Adjective 


adj 


4 


Cardinal Adjective 


Adjective 


adj 


3 


Article and Determiner 


Adjective 


adj 


s 


Nominal Adjective 


Adjective 


adj 


3 


Possessive Adjective 


Adjective 


adj 


3 


Interjection 


Interjection 


Intj 


s 



Variables previously studied 

Several variables used in a number of earlier studies were 
included in the present study. Their presence was essential to provide 
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H of the new 

s, an o insure the adequacy of the answers found to the other 

problems investigated. The abbreviations and descriptions of these 
variables are given below. ^ 

Word frequency index (WOR FRE) Tne word frequency measure 
was obtained by assigning an index number ranging from o to 52 to 
eac wor in each passage used in the study. These index numbers 
were taken from the combined counts given by Thorndike and Lorge 
(1944)- Zero was assigned to words not observed in the word count 

wordTtha'!'* numbers 5I and 52 were assigned to the two groups of 

frequently. This variable was used for Li- 
vidual words and also averaged to find the mean frequencies of words 

twee “nits. Lorge (I948) found a correlation of .5I be- 

*e passje^^ frequency of the words in 

Syllable counts (SYL) The number of syllables was counted to 

S aQ,oW^*rt and sentences, 

of svt^aWe ^ “ coiTelatlon of .69 between the mean number 

of syllables per word in passages and the difficulty of the passages. 

tenre^"'^ (fSSS) found a correlation of .44 between mean sen- 
ength measured in syllables and the comprehension difficulty of 

or semeL syUables in an independent clause 

m sentence IS necessarily related to the number of words in that unit 

This dependence was eliminated by subtracting the number of words 
number of syllables in the independent clause or sentence to 
obtam the net syUables (SYL - WOR). Mean number of syUables per 
sentence was also used as a variable (SYL/WOR). 

Cmnt of words (WOR) Sentence lengths have been most com- 
monly measured by counting the number of words in them. The corre- 
lauon between the mean number of words per sentence and the diffi- 
culty of passages was found to be .29 by Dale and Tyler ( I934 ) , .40 by 

*^^948). This variable hal 
been the best single measure of the grammatical complexity of sen- 

tences and is incorporated in nearly all of the widely used for- 
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Words on the Dale List of j6g Words (DL 769) The Dale List of 
769 Words was compiled by Dale (I93I) from those words which 
were common to both the International Kindergarten Union List and 
to Thorndike’s l,ooo most frequently used words. This variable was 
applied to passages in the present study by determining what propor- 
tion of the running words in a passage appeared on Dale’s list. The 
same word could be counted several times if it recurred within a pas- 
sage. The correlation between the comprehension difficulty of passages 
and this variable was found to be .35 by Dale and Tyler ( I934), .35 by 
Gray and Leary (I935) and .G1 by Lorge (I948). 

Words on the Dale List of 3,000 Familiar Words (DL 3000) The 
words on the Dale list were found (Dale & Chall, I948) to be known 
to 80 per cent of the children in grade four. The list is used, as is the 
shorter one, to find the percentage of the words in a passage which 
appear on the list. Dale and Chall found a correlation of .68 between 
this variable and the difficulties of passages. 

Prepositional phrases (PREP PHR) The PREP PHR variable is 
usually derived by finding the ratio of prepositions to the total number 
of words in the passage. The correlation between PREP PHR and com- 
prehension difficulty was found to be .35 by both Dale and Tyler 
(I934) and Gray and Leary (I935) and .43 by Lorge (I948). It is 
not entirely clear whether the variable measures sentence length or 
whether it measures some other attribute of writing style since sen- 
tences containing prepositional phrases would seemingly be longer 
than sentences which do not contain them. In this case, finding the 
proportions of words appearing in prepositional phrases should result 
in a higher correlation with difficulty than would the proportion of 
prepositions. To study the question, the proportion of words in preposi- 
tional phrases (WOR PREP PHR) was also used as a variable. 

Independent clause (IND CLAU) The sentence, as defined by an 
Initial capital letter and a final punctuation mark, has been the tradi- 
tional unit for analyzing the grammatical complexity of syntactic 
units. Coleman (I962) found some evidence that the independent 
clause is probably a more valid unit of analysis, since a sentence like 
The boy went to the orchard and he picked some apples, may actually 
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be responded to as if it were two separate syntactic units. The con- 
junction and may serve roughly the same function, psychologically, as 
a period. If this is true, measures of syntactic complexity based on the 
independent clause should result in higher correlations than variables 
based on the sentence as it is traditionally defined. In order to study 
this problem, measures of syntactic complexity were found for both 
independent clauses and sentences. 

From the foregoing descriptions of the problems investigated, 
it should be evident that the purpose of the present study was not to 
develop readability formulas for immediate use. To have done so at 
this time would have been poor strategy. In research a single new de- 
velopment often opens new areas for progress. But new developments 
simultaneously present new problems. Readability research has been 
visited not by just one new development but by three. The cloze test 
has been responsible for two. It made possible the development of 
formulas for predicting the readability of units as small as a word and 
it made possible obtaining information essential to the application of 
more powerful and sophisticated mathematical models to the treat- 
ment of readability data. The growing field of psycholinguistics is 
rapidly assimilating he ad hoc theories of readability into a more 
integrated body of theory, joining the fields of psychology, linguistics, 
and literary style, and resulting in new approaches to the measure- 
ment of the various attributes of prose style. As expected, these devel- 
opments have presented new problems for study which must be or- 
dered in priority and studied in a systematic fashion. The author has 
attempted to begin this task. 

Procedure 

The data for the present study were obtained by making cloze 
tests from 20 passages selected to represent a variety of prose styles. 
This provided data on the difficulties of 5,181 words, 405 independent 
clauses, and 365 sentences as well as on the 20 passages. The details 
of how these data were collected are given in the sections that follow. 

Materials 

Passages Twenty passages of between 275 and 300 words each 
were selected from materials used for instructional purposes in the 
areas of literature, history, geography, biology, and physical science. 
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The four passages in each area were selected to provide a roughly 
even distribution in Dale-Chall readability from about the 4.0 to about 
the 8.0 grade levels of difficulty. Care was taken to use materials that 
were not readily available to the subjects, and no two passages were 
by the same author. 

Cloze tests Five cloze test forms were made from each passage 
with each form made by replacing every fifth word with an under- 
lined blank space I5 spaces in length. No deletions were made from 
the first and last sentences of each passage. Generally, a word was 
taken as being defined by the white spaces separating it from other 
words: don’t, U.S.A., 2,182, and re-enter were deleted as single words. 
In hyphenated words like self-made where both parts were free forms, 
each part was deleted separately. Commas, apostrophes, and hyphens 
were deleted along with the rest of the word, except where hyphens 
separated free forms. The five different test forms, forms A through E, 
were made by deleting words 1, 6, 11 , etc., to make form A, words 2, 
7, I2, etc., to make form B, and so on through form E. Every word 
in a passage, excluding those in the first and last sentences, appeared 
as a deleted item in exactly one form. 

Subjects 

The subjects in this study comprised the entire enrollment in 
grades four through eight of the Wasco Union Elementary School 
District in Wasco, California. The sample was chosen for its similarity 
to the general population of school children in the nation with respect 
to standardized achievement test scores, racial and language back- 
grounds, and parents’ occupational status. The subjects were divided 
into five form groups, labeled A through E. Their scores on the Cali- 
fornia Reading Test were used to match the means and distributions 
of reading abilities in the groups. The test had been given by the 
school personnel several weeks prior to the study. The matching pro- 
cedure was done a grade level at a time to insure that an approxi- 
mately equal number of subjects from each grade level was in each 
form group. 

Test administration 

The testing extended over a period of eleven school days. On 
the first day the Stanford Achievement Test: Reading, Form J was ad- 
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ministered and the subjects were given a short cloze est to acquaint 
them with this type of test. For the next ten consecutive school days 
the subjects then took two cloze tests per day. The test periods were 
about 50 minutes long and no subject was stopped before he had 
completed the tests. Each subject took one, and only one, form of the 
cloze test over each passage and all subjects took a test over each of 
the twenty passages. A subject was assigned to a form group and all 
subjects in the same form group took the same cloze test form over 
each passage. The excellent cooperation of the school authorities and 
teachers made it possible to administer the tests under almost ideal 
conditions. 

Scoring 

Final matching of form groups Subjects who failed to take one or 
more of the cloze tests were dropped from the study. The form groups 
were then equated in size by randomly discarding cases from the 
poups with the larger numbers of subjects. There were I35 subjects 
in each of the five form groups. The final groups were then checked 
to see if they were stiU matched in reading ability. The scores on the 
Stanford reading test were used as the dependent variable. The F 
ratio for the between form groups variance shown in Table 2 was not 
significant. An inspection of the means in the cells also showed that 
the matching remained extremely dose. The over-all mean of the group 
was 5.5 in grade placement scores, and the range was from about I.4 
to about I2.7 in each form group. 

Scoring the clcze tests A subject’s response was scored as correct 
when it exactly matched the word deleted to form the ;item. Mis- 
spellings were scored correct if they were intelligible to the scorer and 
not ambiguous. A few ambiguities arose in the case of homonyms, 
each of which could grammatically fit the context of the cloze blank! 
for example there and their. The error rate in scoring was held at or 
below 1 error per 500 items by having a scoring supervisor rescore a 
lo per cent sample of every group of loo tests. In point of fact, the 
error rate seldom reached as high as 1 error per l,ooo items for a 
scorer. When the errors exceeded the prescribed level, the entire set 
was rescored. All scorers were trained to read poor handwriting and 
phonetic spelling. 
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Table 2 Analysis of reading achieve- 
ment scores by cloze test form groups 
and levels of reading ability 



Source 


Degrees of 
freedom 


Mean 

squares 


F 

ratio 


Form Groups 


4 


12.77 


.07 


Ability Levels 


4 


248,223.18 


1,320.37 


Interaction 


16 


96.89 


.52 


Within 


650 


188.02 





Dependent variables 

Measures of comprehension difficulty ( DIF ) The difficulty of each 
word was found by calculating the proportion of subjects responding 
correctly when that word appeared as a cloze item. This was called 
word difficulty. Independent clause difficulty was determined by tak- 
ing the mean of the word difficulties in each independent clause. Sen- 
tence difficulty and passage difficulty were also found by calculating 
the mean of the word difficulties within those units. 

Difficulties at ability levels One of the major problems studied re- 
quired that difficulties be calculated separately for subjects at different 
levels of ability. Thus, the subjects in each form group were ranked 
by the size of their total scores on the Stanford reading test and then 
divided into quintiles. The mean grade placement scores of these 
groups were 3.2, 4.2, 5.I, 6.3, and 8,7. The F ratio for the between 
ability levels in Table 2 shows that this resulted in the formation of 
strata that differed markedly in reading ability. The very small F ratio 
for the interaction of form groups and ability levels shows that groups 
at the same ability level were highly similar regardless of the form 
group in which they appeared. It also may be taken ds evidence that 
the form groups were well matched in the distribution of reading abil- 
ity. The measures of word, independent clause, sentence, and passage 
difficulty were then calculated separately for the groups of subjects at 
each of these levels of ability. 

Analyses and results 

This part of the report has been subdivided into five sections, 
one for each of the five major problems investigated. The method of 
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analysis, the results, and a brief discussion of the major results accom- 
panies each section. 

Linearity of regressions 

Method of analysis The purpose of studying this problem was to 
determine whether linear correlation models are suitable for treating 
the raw data in readability studies. The F test of linearity (Guilford, 
^956) was the method of analysis used. The Pearson product mo- 
ment correlation was calculated for the regression of each linguistic 
variable upon the appropriate measure of comprehension difficulty. 
An estimate of the amount of variance was yielded that could be ac- 
counted for by a fitted straight line. An eta correlation, a measure of 
correlation that is independent of the shape of the regression between 
variables, was also calculated for the same regression. It yielded an 
estimate of the amount of variance that could be accounted for by a 
regression line of any shape. The ratio of the two variances, an F ratio, 
was used to test the hypothesis that there was no difference between 
the amounts of variance accounted for by the linear and non-linear 
correlations. In interpreting these F ratios, it must be understood that 
Pearson product moment correlations can occasionally exceed the eta 
correlations, thereby causing a negative F ratio. 

Word level of analysis Table 3 shows the results of the linearity 
analysis when applied to the regression on word difficulty of the four 
variables derived for each individual word. 

Table 3 Tests of the linearity of 
the regressions of linguistic varia- 
bles on comprehension difficulty at 
the word level of analysis 



Linguistic 

variable 


r 


V 


F* 


SYL 


395 


426 


12.29 


LET 


498 


520 


12.26 


WOR FRE 


286 


334 


13.42 


WOR DEP 


002 


058 


1.34 



* With I4 and 5,166 d.f. and F of I.64, 
P < 05. 



Clearly, all but the word depth variable had a markedly 
curvilinear relationship. In the case of word depth, the correlation was 
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too slight to yield definitive results. The I4 degrees of freedom in this 
calculation occur because the distribution on the dependent variable 
was divided into I5 segments in calculating the eta. 

An examination of the scatter plots of the correlations showed 
that they were similar in siiape. Words containing few syllables or 
letters and the high frequency words tended to be easiest. Small in- 
creases in word length or decreases in frequency resulted in large 
increases in difficulty at first. A gradual leveling off occurred as these 
values continued to change. Examination also showed that a ceiling 
effect may have accounted for a part of the curvilinearity. A number 
of the words were of maximum difficulty, indicated by the fact that no 
subject responded correctly when the word appeared as a cloze item. 
But the curvilinearity was still clearly apparent even when these words 
were ignored. 

Independent clause level of analysis Table 4 shows the results 
when the linearity analysis was applied to the variables quantified in- 
dividually for each independent clause. The curvilinearity was signifi- 
cant only for DL 3000. However, in many other cases the F ratios 
approached a significant level. Of considerable interest is the fact that 
syllables per word and letters per word at this level of analysis showed 
little or no tendency toward curvilinearity. 



Table 4 Tests of the linearity of the regressions of linguistic variables on 
comprehension difficulty at the independent clause level of analysis 



Linguistic 

variable 


r 


V 


F* 


Linguistic 

variable 


r 


V 


F* 


SYL 


-422 


.431 


.49 


DL 3000 /WOR 


443 


.491 


2.89 


SYL/WOR 


-448 


.446 


-.13 


PREP PHR 


-323 


.351 


1.05 


SYL-WOR 


-484 


.501 


1.11 


WOR DEP PHR/WOR 


-210 


.271 


1.57 


LET 


-427 


.434 


.40 


2WOR DEP 


-329 


.334 


.20 


LET/WOR 


-553 


.547 


.47 


SWOR DEP-WOR 


-288 


.294 


.17 


WOR 


-351 


.359 


.35 


SWOR DEP/WOR 


-202 


.257 


1.34 


SWOR FRE/WOR 


298 


.346 


1.76 


IND CLAU FRE 


049 


.115 


.54 


DL 769 /WOR 


421 


.448 


1.47 











* With I4, and 309 d.f, and F of I.96, p - <.05. 



This result is partially explained by the well known fact that normally 
distributed variables tend to exhibit a linear relationship with each 
other where any relationship exists in the first place. While the distri- 
butions underlying the variables were sharply skewed at the word level 
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of analysis, they appealed to be approximately normal when the indi- 
vidual items were averaged^ to form the variables at the independent 
clause level of analysis. This normalizing effect follows almost as a 
matter of necessity from the central limits theorem. 



Passage level of analysis Table 5 shows the results at the passage 
level of analysis. In no case did the F ratio reach the .05 level of sig- 
nificance, but, because of the small number of passages, the test was 
not very powerful at this level. However, there were several indica- 
tions that curvilinearity may have existed in much of the data. The 
most obvious indication is the fact that variables which were first 
quantified at the sentence and independent clause levels and then 
averaged to form variables at the passage level, showed a marked de- 
crease in the sizes of the F ratios. Also supporting this possibility were 



Table 5 Tests of the linearity of the regression of linguistic variables on 
comprehension difficulty at the passage level of analysis 



Linguistic 

variable 


r V F* 


Linguistic 

variable 


r V F* 




Traditional type 
variables 




U. - 

Traditional type 
variables 



SYL/IND CLAU 
SYL/SEN 
LET/WOR 
LET/IND CLAU 
LET/ SEN 
PREP PHR/WOR 
PREP PHR/IND CLAU 
PREP PHR/SEN 
WOR PREP PHR/WOR 
WOR PREP PHR/IND 
CLAU 



-625 


-683 


.72 


WOR PREP PHR/SEN 


-724 


-703 


-800 


-783 


-.35 


2 WOR FRE/WOR 


537 


658 


-696 


-677 


-.25 


DL 769 


676 


743 


-675 


-759 


1.43 


DL 3000 


638 


788 


-807 


-794 


-.28 


WOR/IND CLAU 


-769 


-732 


-673 


-660 


-.16 


WOR/ SEN 


-582 


-569 


-412 


-497 


.51 


2 WOR DEP/WOR 


-546 


-500 


-724 


-755 


.53 


2 WOR DEP/SEN 


-502 


-490 


-753 


-741 


-.21 


IND CLAU FRE/IND 






-456 


-474 


.10 


CLAU 


130 


313 








LET RED* 


350 


644 


-704 


-710 


.08 


LET RED,i 


-377 


-618 



.45 



Form class ratios 



noun/all other, 1 /a 


489 


623 


1.22 


verb/noun, a /1 


127 


237 


.21 


adjective/noun, 3 /I 


-606 


-708 


1.34 


adverb/noun, 4 /I 


098 


463 


1.30 


structural/noun, 5 /I 


-659 


-676 


.21 


noan/verb, 1 /a 


-190 


-300 


.30 


adjective/verb, 3 /a 


-573 


-559 


-.12 


verb/ all other, a/A 


494 


492 


-.01 


adverb/verb, 4 /a 


070 


544 


2.06 


structural/verb, 5 /a 


-564 


-555 


-.07 


noun/ adjective, I /3 


548 


696 


1.79 



Form class ratios 



adverb/adjective, 4/3 


457 


665 


2.10 


structural/adjective, 5/3 


369 


554 


1.23 


noun/adverb, I /4 


-144 


-467 


1.26 


verb/adverb, a /4 


-117 


-513 


1.69 


adjective/adverb, 3/4 


-343 


-495 


.85 


adverb/all other, 4 /A 


228 


538 


1.67 


structural/adverb, 5/4 


-216 


-430 


.85 


noun/structural, I /5 


674 


672 


-.03 


verb/structural, a /5 


666 


629 


-.39 


adjective/structural. 








3/5 


-390 


-548 


1.05 
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Table 5 ( Continued ) 



FoTTii class ratios Form class ratios 



verb/adjective, 2/3 
adjective/all other, 3 /A 


576 

-561 


671 

-650 


1.08 

.93 


adverb/structural, 4/5 
structural/ all 
other, s/A 


366 

-692 


588 

-658 


1.62 

-.40 




Part of speech ratios 


Part of speech ratios 


Noun /all other 


065 


407 


.97 


Noun/preposition 


535 


458 


-.48 


Pronoun/noun 


232 


342 


.36 


Pronoun/preposition 


469 


454 


-.09 


Verb/noun 


200 


427 


.87 


Verb/preposition 


513 


488 


-.16 


Adverb/ noun 


214 


357 


.47 


Adverb/ preposition 


438 


476 


.22 


Preposition/noun 


-458 


-415 


-.23 


Preposition/all other 


-421 


-476 


.32 


Conjunction/noun 


-338 


-473 


.71 


Conjunction/ 








Adjective/noun 


-363 


-516 


.91 


preposition 


-153 


-402 


.83 


Interjection/noun 


613 


693 


1.02 


Adjective/preposition 


211 


494 


1.32 


Noun/pronoun 


-357 


-399 


.19 


Interjection/preposition 


550 


601 


.46 


Pronoun/all other 


393 


394 


.01 


Noun/ conjunction 


564 


606 


.39 


Verb/pronoun 


-357 


-354 


-.01 


Pronoun/ conjunction 


805 


852 


1.41 


Adverb/ pronoun 


-132 


-358 


.63 


Verb/ conjunction 


727 


743 


.26 


Preposition/pronoun 


-442 


-478 


.21 


Adverb/conjunction 


663 


680 


.21 


Conjunction/pronoun 


-524 


-510 


-.10 


Preposition/ conjunction 


215 


356 


.46 


Adjective/ pronoun 


-469 


-461 


-.04 


Conjunction/all other 


-441 


-516 


.49 


Interjection/pronoun 


619 


704 


1.11 


Adjective/conjunction 


304 


526 


1.27 


Noun/verb 


-281 


-417 


.58 


In ter j ec tion /con j unc tion 


607 


685 


.95 


Pronoun/verb 


305 


311 


.02 


Noun/ adjective 


365 


608 


1.88 


Verb/all other 


421 


472 


.29 


Pronoun/ adjective 


392 


391 


-.00 


Adverb/ verb 


115 


405 


.90 


Verb/ adjective 


432 


477 


.26 


Preposition/ verb 


-457 


-535 


.54 


Adverb/ adjective 


471 


493 


.14 


Conjunction/ verb 


-566 


-542 


-.19 


Preposition/adjective 


-112 


-529 


1.86 


Interjection/ verb 


813 


694 


1.02 


Conjunction/ adjective - 


-251 


-401 


.58 


Noun/adverb 


-314 


-426 


.50 


Adjective/all other 


-430 


-489 


.35 


Pronoun/adverb 


-013 


-390 


.89 


Interjection/ adjective 


582 


646 


.68 


Verb /adverb 


-199 


-423 


.85 


Noun/interjection 


533 


695 


1.93 


Adverb/all other 


334 


440 


.51 


Pronoun /inter j ection 


541 


703 


2.00 


Preposition/ adverb 


-392 


-450 


.31 


Verb/interjection 


532 


694 


1.92 


Conjunction/adverb 


-512 


-573 


.49 


Adverb / interjection 


552 


714 


2.08 


Adjective/adverb 


-435 


-490 


.33 


Preposition/interjection 


442 


602 


1.31 


Interjection/ adverb 


626 


714 


1.21 


Con j unc tion /in ter j ec tion 


522 


685 


1.84 










Adjective/interjection 


484 


646 


1.57 










Interjection/all other 


598 


671 


.84 



* With 3 and I 5 d.f. and F of 3.29 p < . 05 . 



the relatively high F ratios associated with DL 3000, LET RED4, and 
LET REDii, none of which were averages of variables quantified at 
lower levels of analysis. 

A third source of evidence was that many of the scatter plots 
gave a distinct visual impression of curvilinearity. Hence, the only 
conclusion that can be drawn from the analysis of the data at this level 
of analysis is that curvilinearity may exist in some of the correlations. 
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A study using a larger number of passages is required to detect it, if 
it does in fact exist. 

Discussion At the word level of analysis, it was clear that the re- 
gressions were curvilinear where a relationship existed in the first 
place. This was not totally unexpected since the distributions of these 
variables were known to be markedly skewed. Tlierefore, it is clear 
that readability formulas designed to predict readabilities of individual 
words must take these curvatures into account. This can be done most 
easily by transforming the numbers obtained from linguistic analyses. 

Transformations would probably also increase the accuracy 
of the readability predictions at the independent clause, sentence, and 
passage levels of analysis. While the F test of linearity showed that 
these regressions did not differ significantly from linearity, these re- 
sults were by no means decisive. That is, being unable to prove that 
the relationships differed significantly from linearity was not identical 
with proving that they were linear. Indeed, several of the relationships 
appeared curvilineai- on visual inspection, but these effects predom- 
inated at the extremes of the distribution. Because there were fewer 
items at these extremes, the F ratio was little affected by these tend- 
encies. Consequently, it would be ill advised not to carefully inspect 
scatter plots when developing a readability formula. 

The finding that DL 3000 was curvilinear at the independent 
clause level of analysis and showed a similar tendency at the passage 
level has a special interest. This is the most heavily weighted variable 
in the Dale-Chall formula, which is the most widely used of the cur- 
rent formulas. A similar tendency was noted in the DL 769 variable. 
When taken together, these results may explain why the formulas have 
been found (Chall, I95®) to be accurate only over limited ranges of 
materials. To test this hypothesis the readabilities of the passages were 
calculated by both formulas and plotted against their comprehension 
difficulties. The results were pronounced S-shaped curves. 

Variable strength as a function of reading ability 

Method of analysis Subjects in each form group were stratified into 
five levels of reading ability; and the difficulties of words, independent 
clauses, and passages were calculated for each ability level separately. 
Using each linguistic vaiiable in turn, language units were stratified 
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into thirds and a three by five analysis of variance was performed. A 
significant interaction effect indicates that the linguistic variable has 
a closer relationship to difficulty for subjects at one level of ability than 
for subjects at another. 

Word level of analysis Table 6 shows the results of the variable 
strength analyses when they were applied at the word level of analysis. 



Table 6 Interaction effects of levels of read- 
ing ability and linguistic variables upon words 
difficulty 



Variable 


F ratio for 
interaction 


Variable 


F ratio for 
interaction 


SYL 


12 . 01 * 


WOR FRE 


12.84 


LET 


16.10 


WOR DEP 


.29 



* Where Fg.sBgn > 2.93. P < .05. 



The interactions between ability and linguistic variable levels were sig- 
nificant for all of the linguistic variables except word depth. However, 
it seems likely that the significant interactions were due to ceiling ef- 
fects rather than to any property of the independent variables them- 
selves. 

A plot of the cell means shov/ed that the interactions were due 
exclusively to low variation among the means of the most difficult third 
of the words. These means were quite low, and there was a large 
number of words to which no subject responded correctly when these 
words appeared as cloze items. The distributions of word difficulties in 
these cells were sharply skewed. The main effects in all analyses were 
significant. 

Word depth failed to show a significant interaction, probably 
due to the fact that its lower correlation with difficulty resulted in the 
words with maximum difficulty being somewhat more evenly distrib- 
uted among the cells. 

Independent clause level of analysis Table 7 shows the results of 
the above analysis applied to individual independent clauses. In the 
independent clause analysis there was no question of ceiling effects 
distorting the results. The difficulties of the independent clauses ranged 
from .08 to .68 and appeared to approximate a normal distribution. 
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Tabh 7 Interaction effects of levels of reading ability 
Sulty variables upon independent clause dif- 



Variable 


F ratio for 
interaction 


Variable 


F ratio for 
interaction 


SYL 

.SYL/WOR 

SYL-WOR 

LET 

LET/WOR 

WOR 

WOR FRE/WOR 
DL 769 


1 . 74 * 

.63 

1.33 

.74 

1.76 

.66 

.74 

1.68 


DL 3000 

PREP PHR/WOR 
WOR PREP PHR/WOR 
WOR DEP 
WOR DEP-WOR 
WOR DEP/WOR 
IND CLAU FRE 


1.30 
1.34 

1.31 
.48 
.45 

1.44 

5.62 



Where Fg jou > a.93, p < .05. 



independent clause frequency exhibited a significant 
Tu Sles significant for 



assage level of difficulty Table 8 shows the results of analysis 

oalr;t",:r 

cause of the labor involved in calculations and beeause of the regu- 

Table a Interaction effects of levels of reading abiUty and Unguis- 
tic vanables upon passage difficulty ^ 



Variable 



SYL/WOR 

SYL/IND CLAU 

SYL/SEN 

LET/WOR 

LET/IND CLAU 

LET/SEN 

PREP PHR/WOR 

PREP PHR/IND CLAU 

PREP PHR/SEN 

WORPREPPHRyV/OR 

WOR PREP PHR. IND CLAU 

WOR PREP PHR/SEN 



F raUo for 
interaction 



Variable 



F ratio for 
interaction 



1.40* 


WOR FRE/WOR 


.49 


DL 769 /WOR 


.72 


DL 3000 /WOR 


1.37 


WOR/IND CLAU 


1.41 


WOR/SEN 


1.72 


WOR DEP/WOR 


1.20 


WOR DEP/SEN 


.59 


IND CLAU FRE/.IND CLAU 


.49 


LET RED 4 


1.28 


LET REDii 


.59 


pn/conj 


1.44 


adj/v 



Where Fg.,, > 3.00, p < .05. 



.43 

1.41 

1.43 

1.37 

1.36 

.54 

.37 

2.14 

1.19 

2.20 
1.26 

.35 



lanty m the results, the analysis was applied to only two of the part of 

speech ra 10s. None of the interactions at this level of analysirwlre 

significant, but the mam effects associated with aU of the independent 
variables were significant. muepenaent 
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Discussion The primary purpose of investigating this problem was 
to find out if the same readability formulas could be validly used for 
predicting difficulty for subjects having both high and low reading 
abilities. Evidently they can. Although the analysis at the word level 
was indecisive because of ceiling effects, the results at the other levels 
of analysis showed rather clearly that the correlations between linguis- 
tic variables and difficulty do not change as a function of reading 
ability. 

The one exception to this generalization is independent clause 
frequency. This interaction may have arisen because the variable was 
based upon the utterances of elementary school children. Such chil- 
dren could be expected to use only a limited range of grammatical 
patterns, and these would probably be the most frequent patterns in 
the language. Variations in the frequencies of patterns within the 
sample of language might be expected to have some correlation with 
the difficulty of the patterns for young children. But with older chil- 
dren all or most of these patterns would be likely to be over-learned. 
Hence, this scale in its present form may not extend over a wide 
enough range to make independent clause frequency of much value 
for predicting difficulty for any but very young children. In view of its 
very low correlations with difficulty, it is doubtful if the variable in its 
present form is of much use even with very young children. 

Predicting difficulties of small language units 

Purpose In this part of the study one intent was to find out if it is 
possible to predict the difficulties of individual words, independent 
clauses, and sentences with enough accuracy to make the development 
of special formulas of practical interest. 

Method of analysis The stepwise multiple regression technique 
used in this analysis involved adding one variable at a time to the 
multiple regression equation and computing a new multiple correla- 
tion with comprehension difficulty after each new variable was added. 
Following each step, a partial correlation was calculated between each 
of the variables not yet added to the equation and comprehension dif- 
ficulty. The variables already in the equation were held constant in 
these partial correlations. The new variable added at each step was 
the one with the highest partial correlation with comprehension dif- 
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ficulty. Tlie procedure was halted when there was no longer a variable 
having a significant partial correlation. A predicted difficulty was then 
computed for each language unit in the sample and these were sub- 
tracted from the actually observed difficulties to obtain a set of re- 
siduals. An inspection of a plot of the residuals reveals curvilinearity 
in the multiple regression. 

It was planned that the analysis would begin with the calcu- 
lation of a multiple R and the residuals for the data at each level of 
analysis. If the multiple correlations had been very low and if the 
plots of the residuals had indicated that curvilinearity was present, 
appropriate transformations would have been performed on the lin- 
guistic variables and the calculations repeated. As indicated below, the 
multiple correlations were so high that the second step was not 
necessary. 



Word level Table 9 shows the matrix of correlations among the 
variables derived at the word level of analysis. In interpreting these 



Table 9 Intercorrelations among the variables at the word 
level of analysis 



Variable 


X 


s 


2 


3 


4 


5 


1 . COM DIF 


29.66 


21.61 


-395* 


-498 


286 


002 


a. SYL 


1.41 


.76 




807 


-487 


-082 


3 . LET 


4.47 


2.27 






-482 


-115 


4 . WORFRE 


46.88 


13.17 








071 


5 . WORDEP 


1.65 


.76 











* Where d.f. = 5>181 and r > .oa6, p < .05. 



correlations it should be remembered that none of these variables is 
linearly related to word difficulty and, in most cases, the linear corre- 



Table 1 o Stepwise multiple regression analy- 
sis of the correlations between linguistic vari- 
ables and word difficulty 



Step 


Variable 

entered 


F* to 
enter 


R 


R2 


Increase 
in R2 


1 


LET 


1708.14 


.498 


.248 


.248 


2 


WOR DEP 


21.80 


.501 


.251 


.003 


3 


WORFRE 


19.70 


.504 


.254 


.003 


4 


SYL 


3.89 


.505 


.255 


.001 



♦ Where F = 3.84, p < .05. 
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lations are probably somewhat lower than the curvilinear correlations 
among the variables. 

The correlations were analyzed using the stepwise multiple 
regression technique. Table lo shows the results of that analysis. The 
multiple correlation with difficulty was .05 when all variables had 
been entered. 

An inspection of the plot of residuals showed that the corre- 
lation would have been considerably higher had the linguistic variables 
been first transformed to make their relationships to word difficulty 
hnear. Of special interest was the behavior of word depth. When let- 
ters were held constant, the correlation with word difficulty increased 
from .002 to .06, suggesting that at the word level of analysis word 
depth acts as a suppressor variable. Further interpretation is difficult 
because of lack of hnearity in the correlations. 

Independent clause level Table 11 shows the correlations among 
the variables derived at the independent clause level of analysis. 
While most of these relationships were linear, it is not known how 
the curvilinearity underlying many of these variables may have af- 
fected the sizes of these correlations. 

Even so, some interesting relationships appeared. First, the 
correlations with independent clause difficulty were generally high. 
All but one, involving independent clause frequency, were significant. 



Table 11 Intercorrelations among the variables at the independent clause level of analysis 



Variable 


X 


« 2 


3 


4 


5 


6 


00 

(0 

0 


11 


Iz 


13 


U 


I5 16 


1. INDCLAUDIF 


32.54 


11.98 -42* 


-45 


-48 


-43 


-55 


-35 30 42 44 


-32 


-21 


-33 


-29 


-2o 05 


2. SYL 


17.99 


13.34 


35 


91 


99 


28 


97 -20 -33 -29 


77 


30 


91 


80 


27 -07 


3. SYL/WOR 


1.39 


.26 




64 


29 


80 


16 -39 -53 -62 


18 


13 


10 


05 


00 05 


4. SYL-WOR 


5.30 


5.21 






88 


52 


78 -29 -46 -47 


68 


29 


71 


60 


18 -04 


5- LET 


56.82 


41.88 








28 


98 -18 -30 -26 


76 


29 


92 


82 


28 -06 


6. LET/WOR 


4.42 


.70 










11 -33 -46 -54 


13 


09 


07 


03 


-02 09 


7. WOR 


12.69 


8.86 










-13 -22 -16 


75 


28 


96 


86 


30 -08 


8. 22WORFRE/WOR 


47.25 


4.47 










59 71 


-14 


-07 


-10 


-07 


-02 -01 


9. DL769/WOR 


.71 


.17 










72 


-28 


-20 


-17 


-12 


-05 06 


lo. DL 3000/ WOR 


.85 


.13 












-23 


-17 


-11 


-06 


01 00 


11 , PREPPHR 


.37 


1.51 














65 


68 


57 


14 -09 


I2. WOR PREPPHR 


.29 


.25 
















19 


11 


-04 -04 


I3. 2WORBEP 


20.96 


18.95 


















97 


50 -08 


14 - 2 WORDEP-WOR 


8.27 


10.88 




















62 -08 


I5. 2WORDEP/WOR 


1.56 


.41 




















-09 


16 . IND CLAU FRE 


162.82 


264.77 























“ Where d.f. = 405 and r > .lo, p < .05. 
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The results of the multiple regression analysis of the variables 
are shown in Table I2. The size of the multiple correlation, .665, ex- 
ceeded the validity correlations reported for several widely used read- 
ability formulas. 



Table 1 2 Stepwise multiple regression analysis of 
the correlations between linguistic variables and 
independent clause difficulty 



Step 


Variable 

entered 


F* to 
enter 


R 


H2 


Increase 
in Ra 


1 


LET/WOR 


177.23 


.553 


.306 


.306 


2 


WOR 


56.25 


.625 


.391 


.085 


3 


DL 3000/WOR 


13.23 


.640 


.410 


.019 


4 


WOR DEP/WOR 


12.83 


.655 


.429 


.019 


5 


WOR PREP PHR 


5.45 


.660 


.436 


.007 


6 


SYL 


4.62 


.665 


.442 


.006 



* Where F = 3.86, p < .05. 



In addition, an inspection of the plot of residuals showed that 
there was a definite tendency toward curvilinearity. This result was not 
altogether unexpected since DL 3000 had shown a significant degree of 
curvilinearity in the earlier analyses. On the other hand, the extent of 
this curvilinearity seemed to be greater than was expected to result 
from DL 3000 alone. Of major interest also was the large number of 
variables entering the equation which seemed to suggest that a num- 
ber of style factors influence comprehension difficulty. Another point 
of interest was that many of the variables entering the equation were 
variables being used in this study for the first time. This suggests that 
future improvements in readability formulas can accrue as much from 
the development of new variables as from the use of improved analytic 
and research techniques. Finally, the fact that none of the variables 
stood out as being of superior importance was important. With shght 
chance variations in the data, another set of variables could have en- 
tered the equation. 

Sentence level of analysis Although all of the analyses reported 
in the present study were performed at both the independent clause 
and sentence levels of analysis, only the analyses at the independent 
clause level have been reported because the findings at these two levels 
were almost identical since over 90 per cent of the sentences were also 
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classed as independent clauses and since almost the identical set of 
variables was used at both levels. At the sentence level, the multiple R 
was of interest since it was somewhat higher than at the independent 
clause level of analysis, .680. Again, the plot of the residuals showed 
some evidence of curvilinearity. 

Discussion The purpose of the analyses was to find out if readabil- 
ity formulas could be designed to make useful predictions of difficulty 
for individual words, independent clauses, and sentences. In the past 
readability formulas have received widespread use and have had va- 
lidities ranging from .5 to .7. If these validities may be taken as a cri- 
terion of usefulness, then the formulas represented by the multiple 
correlations in this analysis must be said to be useful. However, it 
was obvious that the correlations can easily be increased by several 
methods. First, the curvilinearity of the predictions can be eliminated 
by performing the appropriate transformations on the variables; and, 
second, it was evident that the best possible linguistic variables for use 
at these levels are far from being discovered. Only four variables were 
explored at the word level and only I5 at the independent clause and 
sentence levels. The parts of speech, aside from the preposition, have 
yet to be tried, and it is obvious that other types of variables should 
be tried. 

Validities of readability formulas 

Analysis The analysis portion of the study was designed to deter- 
mine if the use of new linguistic variables and modem testing tech- 
niques could improve the ability to predict passage difficulty. No effort 
was made to transform the linguistic variables or to do any of the 
other tasks necessary for making predictions with maximum accuracy. 
Instead, the variables quantified at the passage level were entered di- 
rectly into a stepwise multiple regression analysis exactly as in the 
preceding section. 

At this point, there were too many variables to be handled 
simultaneously by the computer program used, so they were analyzed 
in two sets. The first set consisted of the 47 variables shown in 
Table I3. 

These were the traditional type variables numbered 1 to 22 
and the form class ratios numbered 23 to 47 in Table I3. Intercorre- 



TABLE 13. INTERCORRELATIONS AMONG PASSAGE DlFFICUtTY ANO THE FORM CUSS AND TRADlTlONftL TYPE VARIABLES (Wg20l 



N«me of 
Varlabitt 



3 4 5 6 7 8 9 10 II 12 13 14 15 



1 PAS oir 

2 SYl/WOR 

3 syl/ino clau 

4 SYL/SEN 

5 LET/WOR 



29.66 

1.42 



9.06 

0.16 



-63 



21.71 

25.48 

4.47 



10.23 

14.06 

0.40 



•80 -70 -68 -81 -67 -41 -72 -75 -46 -70 ./2 54 68 i 
61 28 93 57 22 65 64 50 74 66 51 -47 -87 

80 57 too 76 47 93 90 54 92 89 -48 -65 -! 

28 82 too 15 64 88 24 65 87 -27 -4l 

55 24 62 59 48 65 58 47 - 39 - 79 -C 



6 LET/INO CLAU 

7 LET/SEN 

8 PREP PHR/woR 

9 PREP PHRAnO clau 
10 PREP PHR/SEN 



68.20 

80.43 

O.ll 

1.67 

9.91 



31.28 

45.13 

0.03 

1.00 

1.11 



79 44 91 91 50 89 88 -45 -61 -5 

It 60 85 19 59 84 -23 -36 

73 58 95 71 55 -44 -67 -E 

91 76 98 90 - 52 - 70 -E 

62 90 98 -43 -63 -5 



11 WOR PREP PHRA^OR 

12 WOR PREP PHR/INO CUU 

13 WOR PREP PHR/SEN 

14 7W0R PRE/WOR 

15 OL 769 



Oo33 

5.18 

5.93 

46.88 

0.69 



0.11 

3.36 

3.75 

2.35 

0.10 



79 65 -52 .72 
91 -55 -71 -E 
-45 -63 -5 
63 E 
e 



16 OL 3000 

17 W0R/5EN 

18 wor/ino CUU 

19 TWOR OEP/WOR 

20 rwOR OEP/SEN 



0.84 

17.92 

15.04 

1.65 

31.45 



0.09 

10.44 

6.38 

0.23 

24.85 



21 INO CLAU PRE/INO CUU 

22 i.ET RED. 

23 LET REO„ 

24 Noun/alf other « 1/A 

25 Vtrb/noun« 8/1 



185.74 

0.34 

0.46 

0.31 

0.47 



93.33 

0.30 

0.26 

0.04 

O.fl 



26 Adjective/noun, 3/1 

27 Adverb/noun« 4/1 

28 Structural/noun« 5/1 

29 Noun/verb« 1/2 

30 Verb/all other« 2/A 



0.31 

6.11 

1.35 

2.25 

0.14 



0.12 

0.08 

0.24 

0.68 

0.03 



31 Adject! ve/verb, 3/2 

32 Adverb/verbf 4/2 

33 Structural/verbf 5/2 

34 Noun/adject !ve« 1/3 

35 Verb/adjact!ve, V3 



0.71 

0.23 

3.00 

3.82 

1.80 



0.35 

0.15 

0.90 

2.15 

0.99 



36 Adjectfvj/al I oiJier, 3/A 

37 Adverb/adject!ve« 4/3 

36 Structural/adject!va« 5/3 

39 Noun/adverb« 1/4 

40 Verb/adverb, 2/4 



0.09 

0.43 

4.87 

18.80 

8.01 



0.03 

0.39 

1.87 

18.78 

7.59 



41 Adject fve/adverb« 3/4 

42 Adverb/all other« 4/A 

43 Structural/advetb« 5/4 

44 Noun/fttructura1« 1/5 

45 Verb/atructural« 2/5 



5.96 

0.04 

25.52 

0.77 

0.36 



6.46 

0.02 

28.35 

0.17 

0.09 



46 

47 

48 



Adjactf ve/s true turalf 3/5 
Advarb/structural « 4/5 
Structural/al I others 5/A 



0.23 

0.09 

0.41 



0.07 

0.07 

0.04 
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19 


20 


21 


22 


23 


24 


25 


26 


27 


28 


29 


30 


31 


32 


33 


34 


35 


36 


37 


38 


39 


40 


41 


42 


43 


lA 


45 


46 


47 


48 


1 -77 


-55 


-50 


13 


35 


-38 


49 


13 


-61 


10 


-66 


-19 


49 


-57 


07 


-56 


55 


58 


-56 


46 


37 


•14 


-12 


-34 


23 


-22 


67 


67 


-39 


37 


-69 


1 


-08 


-03 


-07 


-24 


13 


-08 


«55 


48 


-53 


25 


58 


-72 


73 


-41 


72 


-52 


-68 


60 


-59 


•48 


34 


25 


57 


-55 


38 


-33 


-73 


47 


-58 


51 


' 98 


43 


57 




-36 


24 


-33 


-29 


50 


-09 


51 


37 


-59 


61 


00 


65 


-49 


-57 


48 


-38 


-39 


16 


to 


34 


-18 


24 


-51 


-66 


32 


-28 


59 


r 86 


81 


94 


14 


-34 


23 


-48 


02 


50 


19 


58 


13 


-35 


44 


15 


43 


-49 


-48 


42 


•22 


-37 


00 


-04 


16 


05 


08 


-55 


-46 


29 


-06 


48 


> 4l 


-07 


-01 


03 


-27 


II 


-07 


-46 


38 


-*?7 


23 


49 


-61 


59 


-37 


61 


-40 


-54 


48 


-57 


-33 


30 


22 


43 


-52 


31 


-33 


-65 


36 


-56 


49 


1 99 


51 


61 


06 


-37 


24 


-35 


•24 


47 


-05 


52 


32 


-54 


56 


04 


61 


-47 


-54 


45 


-36 


-36 


13 


08 


29 


-14 


20 


-52 


-63 


29 


-24 


58 


) 84 


83 


96 


15 


-34 


22 


-49 


08 


47 


23 


57 


07 


-29 


38 


18 


37 


-46 


-43 


37 


-19 


-34 


-03 


-06 


11 


09 


04 


-54 


-41 


25 


-03 


45 


35 


-16 


-09 


21 


-17 


-09 


14 


-70 


15 


-57 


16 


63 


-74 


53 


-41 


74 


-11 


-34 


26 


-51 


-03 


56 


46 


57 


-58 


55 


•22 


-76 


09 


-61 


58 


J 87 


32 


39 


14 


-32 


12 


-17 


-50 


39 


-24 


41 


53 


-72 


64 


-09 


78 


-37 


-52 


42 ^2 


-28 


34 


24 


45 


-30 


39 


•42 


-76 


24 


-37 


62 


t 90 


60 


71 


22 


-35 


14 


-31 


-33 


46 


-07 


51 


41 


-63 


61 


00 


70 


-43 


-53 


45 


-37 


-32 


25 


17 


37 


-18 


31 


-50 


-70 


27 


-27 


61 


' 41 


-07 


00 


19 


-21 


02 


10 


-78 


28 


-54 


20 


74 


-87 


68 


-36 


86 


-21 


-46 


38 


-52 


-15 


57 


45 


65 


-55 


59 


-26 


-85 
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lations were calculated among the variables yielding a rather massive 
correlation matrix. Data were then entered into the stepwise multiple 
regression analysis. The results, which are shown in Table I4, were 
somewhat surprising since the multiple correlation was markedly 
higher than any that has yet been reported in any known readability 
study. 



Table I4 Stepwise multiple regression analysis of 
correlations between passage difficulty and the tra- 
ditional type and form class ratio linguistic vari- 
ables 



Step 


Variable 

entered 


F* to 
enter 


A 


R2 


Increase 
in Ra 


1 


LET/IND CLAU 


33.61 


.807 


.651 


.651 


2 


n/str (1/s) 


6.04 


.862 


.743 


.092 


3 


LET/WOR 


5.50 


.899 


.808 


.065 


4 


SWOR DEP/WOR 


7.59 


.934 


.872 


.064 



♦ When F > 4.38, p < .05. 



A similar analysis was performed on the ratios derived from 
the counts of words falling into the traditional part of speech cate- 
gories. The variables are shown in Table I5. 

First, a correlation matrix was calculated and then the cor- 
relations were used to calculate the multiple regression analysis shown 
in Table 16. 



Table 16 Stepwise multiple regression 
analysis of correlations between passage 
difficulty and parts of speech linguistic 
variables 



Step 


Variable 

entered 


F* to 
enter 


R 


R2 


Increase 
in R2 


1 


pn/conj 


33.07 


.805 


.648 


.648 


2 


prep/n 


4.27 


.845 


.719 


.071 



♦ When F > 4.38, p < .05. 



Although only one variable actually entered the calculation, 
the size of the R agreed fairly well with the one obtained from the 
analysis of other sets of variables. The ratio of prepositions to nouns 
was allowed to enter this equation in order to gain some idea of how 
this set of variables would behave if the study were repeated using a 
larger number of passages. 
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Discussion It is clearly evident that the present level of precision 
in readability prediction ,n be markedly increased, possibly even 
over the levels reached by the predictions shown in Tables I4 and 16. 
Improvements will probably result from the development of more pow- 
erful bnguistic variables, evidenced by the fact that the variables enter- 
ing the multiple correlation equations were, almost without exception, 
newly developed. 

It is apparent that available readability formulas contain too 
few variables and thereby ignore variables of prose style which can 
make important independent contributions to the multiple correlations. 

New linguistic variables 

Word depth Word depth seemed a very useful measure. Although 
its linear correlation at the word level of analysis was not significant, 
its eta correlation at this level, .058, was significant and accounted for 
a significant amount of variance in the analysis of variance. Further, 
as seen in Table 9, it appeared to act as a suppressor variable in the 
multiple correlation with word difficulty. Because of its skewed dis- 
tribution, the usefulness of the measure could probably be improved 
by transforming the word depth numbers. 

At the independent clause level of analysis, mean word depth 
appeared to provide a measure of grammatical complexity not closely 
related to measures of the length of independent clauses. This is indi- 
cated by its relatively low correlation with counts of the numbers of 
letters, syllables, and words in independent clauses shown in Table 11. 
As a result, this variable contributed substantially to the multiple cor- 
relation with independent clause difficulty (Table I2) even after a 
measure of length was already in the equation. The two other variables 
based upon word depth exhibited higher correlations with difficulty, 
but they were also more highly correlated with measures of length. 

At the passage level, essentially the same relationships ob- 
tained except that mean word depth showed a closer relationship to 
measures of sentence length than to measures of independent clause 
length. Mean word depth’s correlation with syllables per independent 
clause was .48; with syllables per sentence, .81; with letters per inde- 
pendent clause, .5I; with letters per sentence, .83; with words per 
independent clause, .59; and with words per sentence, .86. This sug- 
gested that mean word depth and measures of sentence length tended 
to measure grammatical complexity, while measures of independent 



TAILE 15i. 



INTEKCORKELATIONS AMONG PASSAGE OIFFICUITV ANO EACH OF THE PANT OF SPEECH PATIOS, (Nb20) 



V«rf 

«blM 



5 6 7 8 9 10 II 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 



1 

2 

3 

4 

5 

6 

7 

8 
9 
10 

If 

12 

13 

14 

15 

16 

17 

18 

19 

20 

21 

22 

23 

24 

25 



07 23 20 .21 >46 >34 >36 6l >36 39 *36 >13 <44 >52 *47 62 >28 30 42 12 >46 *57 •‘*9 61 “3t .01 .20 33 .39 .51 .44 

.73 .82 .66 .19 *65 .61 13 70 *57 49 02 58 30 47 12 73 *54 *42 *32 53 *02 34 13 66 *09 23 *47 47 19 37 

84 59 .16 58 09 09 *68 95 -75 *31 *67 *41 *68 09 *66 89 59 20 *63 *C3 *60 09 *54 40 *12 43 *48 *08 *53 

57 .12 64 15 02 *73 77 -48 *26 *71 *41 *69 02 *86 55 83 07 *79 *12 *75 02 *57 29 -03 41 *49 *12 *51 

*39 54 31 30 .52 50 *53 49 *56 *21 *47 31 .48 53 28 84 *55 09 *32 30 *86 .41 *74 95 *80 *56 *75 

.12 42 *63 12 *28 18 *25 31 05 25 *62 08 .18 *30 *35 49 *03 34 *63 33 25 39 *53 54 25 50 

22 .29 *27 41 .26 08 *28 26 *24 *30 *36 34 29 21 *34 64 *26 *29 *34 10 .14 32 *31 30 *30 

*31 .11 *13 02 25 .01 01 21 *30 *17 01 *27 27 05 08 40 *31 *28 *28 *24 16 *10 *19 20 

.15 24 .2! 20 .24 *26 *21 100 *09 19 22 37 *33 *34 *23 100 *27 *18 *29 52 *32 *38 *31 

*75 76 32 98 84 94 *15 92 *67 .72 *22 86 50 78 .I5 67 *27 10 *48 59 47 60 

*83 .40 *76 *55 *80 24 *67 92 67 16 *70 .18 *71 24 *52 48 *07 42 *50 *13 *59 

25 76 58 79 *21 50 *93 *35 *40 51 17 48 *21 57 .29 27 *51 53 31 60 

26 39 37 20 31 *27 *42 76 19 34 42 20 *38 *82 *76 58 .40 *38 *30 

82 95 *25 89 *67 .74 .27 92 47 8t .24 71 *21 17 *55 68 50 68 

80 .27 74 *50 *58 *06 68 86 65 *26 46 *25 *02 *27 40 62 42 

.21 83 *70 *78 .18 84 46 89 *21 59 *33 07 *46 57 41 69 

.09 19 21 37 *33 *35 *23 100 *28 *18 *29 52 *32 *38 *31 

.48 .85 .07 91 48 83 *09 61 *27 >03 *40 51 37 50 

36 36 .49 *10 >Jt6 19 *54 36 *24 47 *51 *21 >56 

•17 .87 "43 .94 21 *39 37 18 25 *39 *16 *49 

•|8 14 07 37 >74 *63 .89 90 *72 *62 *65 

42 88 .33 66 .15 12 *54 67 42- 65 

44 .34 18 .13 .15 .03 13 59 14 

.23 42 .38 .13 .30 44 23 60 

.27 .18 *29 52 .32 .38 .3 i 



TABLE 15b, HAHES OF THE VARIABLES IN TABLE 15a, 
THEIR HEANS, ANO STANOARO OEVIATIONS. 



26 

27 


No. 


Names of 
Variables 


X 


s 


28 










29 


1 


Passage Difficulty 


26.66 


9.06 


30 


2 


Noun/all other 


• 26 


•05 


3 


Pronoun/noun 


• 31 


.18 


31 


4 


Verb/noun 


• 75 


•24 


32 


5 


Adverb/roun 


•23 


• 14 


33 










34 


6 


Prepositlon/noun 


.50 


.11 


35 


7 


ConJ unc t lon/noun 


.29 


• 18 




8 


AdJectI ve/noun 


.90 


.22 


36 


9 


Interject lon/noun 


.01 


.01 


37 

38 

39 


10 


Noun/pronoun 


.47 


3.48 


II 


Pronoun/all other 


•07 


•03 


40 


12 


Verb/pronoun 


2.79 


•96 




13 


Adverb/pronoun 


.79 


.34 


41 


14 


PreposI tion/pronoun 


.2.30 


1.90 

.94 


42 

43 

44 


15 


ConJ unc 1 1 on/pronoun 


1.14 


16 


Ad Jec 1 1 ve/pronoun 


3.95 


2.90 


45 


17 


interject lon/pronoun 


.01 


• 03 




le 


Noun/verb 


1.50 


.65 


46 


19 


Pronoun/verb 


.40 


.12 


47 


20 


Verb/all other 


.18 


.04 


48 








49 


21 


Adverb/verb 


.30 


• 15 


50 


22 


Prepos It lon/verb 


.76 


.38 




23 


Conjunction/verb 


.40 


.22 


51 


2k 


Adjective/verb 


.32 


.57 


52 


25 


Interjection/verb 


.01 


.02 


53 










54 


26 


Noun/adverb 


6.25 


3.88 


55 


27 


Pronoun/adverb 


1.60 


1.02 




26 


Verb/adverb 


4.20 


2.21 


56 


29 


Adverb/all other 


•06 


•03 


57 


30 


PreposI tion/adverb 


3.29 


2.39 


58 








59 


31 


ConJ unc 1 1 on/a^^A irb 


1.59 


t.02 


60 


32 


AdjectIve/adverL) 


5.41 


3.47 




33 


Interjection/adverb 


• 01 


•03 


61 


34 


Noun/prepositlon 


2.15 


.88 


62 

63 


35 


Pronoun/preposI tion 


.69 


.51 


64 







TABLE 15b. (Continued) 



38 76 *87 95 72 86 
78 .51 40 56 26 
.81 79 66 67 
*85 *68 .80 
69 92 

61 



No. 


Names of 
Variables 


X 


s 




36 


Verb/prepositlon 


1.63 


.85 




37 


Adverb/preposI tIon 


.53 


.50 




38 


PreposI tion/ell other 


.13 


.03 




39 


Conjunction/preposition 


.62 


.44 




40 


AdJectI ve/preposi tion 


1.86 


.57 




41 


Inter Jec tion/preposi tion 


.02 


.07 




42 


Noun/conjunctlon 


4.64 


2.31 




43 


Pronoun/con J unction 


1.30 


.79 




44 


Verb/conJ unc tion 


3.26 


1.66 




45 


Adverb/conjunct Ion 


1.00 


.83 




46 


Preposition/conjunction 


2.32 


1.25 




47 


ConJ unc tl on/a 1 1 other 


.07 


•03 




48 


Adject 1 ve/conj unc tion 


4.06 


2.07 




49 


In ter Jectlon/conJ unction 


.03 


.11 




50 


Noun/adJectIve 


1.19 


• 39 




52 


Pronoun/adJectIve 


.37 


.22 




52 


Verb/adject Ive 


.83 


• 34 




53 


Adverb/adject Ive 


.26 


• 15 




54 


PreposI tion/adjec t Ive 


.58 


cle 




55 


^njunctlon/adjectlve 


.33 


• 19 




56 


Adjective/all other 


•23 


.05 




57 


Interject lon/adjective 


.01 


.02 




.58 


Noun/Interjectlon 


}.ol 


13.39 




59 


Pronoun/Interjectlon 


1.34 


4.64 




60 


Verb/Interjectlon 


2.95 


10.38 




61 


Adverb/Interjectlon 


1.30 


4.45 




62 


Preposition/Interjection 


1.76 


7.15 




63 


Conjunction/Interjection 


.45 


1.61 




64 


AdJectI ve/I nterjectlon 


3.34 


12.63 




65 


Interjectlon/all other 


.01 


.01 
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47 


48 


49 


50 


51 


52 


53 
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55 


56 


57 


58 


59 


60 


61 


62 


63 


64 


65 


63 


<:4 


47 


51 


44 


-42 


-15 


21 


55 


56 


81 


73 


66 


22 


-44 


30 


61 


37 


39 


43 


47 


-11 


-25 


-43 


58 


53 


54 


53 


55 


44 


52 


48 


60 


11 


21 


-41 


-31 


-29 


44 


-56 


-34 


16 


49 


-06 


15 


-03 


28 


-43 


17 


13 


64 


-37 


-16 


-38 


45 


-39 


03 


15 


-09 


-09 


-10 


-OB. 


-14 


-10 


-12 


14 


09 


12 


79 




42 


-62 


56 


25 


18 


-40 


28 ' 


-07 


09 .44 


39 


•38 


09 


-10 


85 


57 


52 


-21 


53 


-51 


08 


06 


07 


06 


07 


05 


06 


06 


09 


.02 


06 


63 


6i 


.38 


-62 


60 


25 


02 


-43 


15 


-03 


02 


-44 


45 


•39 


02 


-17 


69 


66 


49 


-23 


56 


-49 


02 


03 


03 


03 


03 


03 


03 


03 


02 


31 


34 


65 


62 


80 


-75 


65 


66 


28 


-22 


27 


03 


46 


1 

VO 


35 


•16 


30 


-31 


33 


17 


88 


-67 


30 


-22 


29 


24 


25 


24 


25 


19 


24 


22 


30 


-61 


-90 


-65 


-77 


-74 


79 


.40 .62 -67 


-09 


-25 


-15 


-40 


52 


-14 


28 .63 •()8 


-37 


•43 -68 


44 


-21 


45 


.65 


-11 


-12 


-11 


-14 


03 


-09 


-03 


-64 


-30 


-08 


36 


31 


27 .48 


95 


23 


-25 


.87 -47 


-69 


•43 


-78 


96 


-78 


-29 


-23 


39 


29 


34 


-23 


.90 


-29 


-27 


-28 


-29 


-28 


-29 


-24 


-28 


-26 


-28 


-30 


-43 


-15 


-22 


-04 


03 


12 


39 


-33 


-25 


-18 


•19 -07 


09 


08 


28 


-31 


-92 


-38 


-60 


-10 


-58 


-19 


75 


-32 


-03 


-04 


-03 


-05 , 


04 


-03 


01 


-31 


100 


87 


61 


70 


75 


-52 


-06 


57 


98 


55 


74 


65 


83 


00 


-35 


28 


100 


27 


27 


26 


65 


-40 


-25 


-27 


99 


53 


55 


53 


57 


35 


50 


43 


100 


-15 


-14 


-56 


-52 


-38 


56 


-30 


-24 


.-13 


09 J*5 


-25 


-28 


14 


-07 


06 


-15 


07 -62 


-55 


-51 


21 


-23 


41 


-14 


-14 


-14 


-14 


-14 


-12 


-14 


-13 


-14 


24 


29 


84 


65 


46 


-65 


44 


. 22 


24 


-23 


45 


09 


21 


-39 


25 


i*33 


24 


12 


95 


71. 


56 


-13 


47 


-65 


24 


10 


10 


10 


11 


05 


09 


07 


24 


-21 


-21 


-64 


-43 


-42 


51 


-31 


-20 


-19. 


•12 


-49 


-10 


-31 


21 


-13 


14 


-21 


-03 


•7* 


-41 


-54 


16 


-28 


45 


-20 


-1B 


-1B 


-1B 


-19 


-15. 


-10 


-16 


-20 


20 


20 


-08 


04 


4l 


-1B 


19 


44 


18 


06 


-10 


-05 


36 


-09 


08 


12 


20 


-29 


•45 


-45 


40 


-52 


-12 


31 


19 


16 


16 


16 


17 


13 


16 


14 


20 


•25 


-30 


-64 


-63 


-49 


68 


-35 


-33 


-23 


06 - U 7 


-27 


-34 


23 


•09 


11 


-24 


-05 -67 


-62 


-61 


26 


-27 


49 


-24 


-15 


-16 


-15 


-16 


-11 


-15 


-13 


-24 


-27 


-16 


-38 


-34 


ip22 


29 


21 


-09 


-23 


•39 .68 .62' 
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Clause length tended to measure something else, perhaps just 
length. 

Letter redundancy The results for the letter redundancy variables, 
LET RED4 and LET REDii, were difficult to interpret. The first prob- 
lem was that short passages, consisting of only about 1,500 letters 
each, were used. Carterette and Jones (I963) showed that passages of 
about 10,000 letters were required to obtain stable estimates of letter 
redundancy. The second problem was that the Pearson product mo- 
ment and eta correlations with passage difficulty contradicted each 
other. Table 5 shows that the Pearson product moment correlations 
were .35 and —.38 for LET RED4 and LET REDn, respectively. Nei- 
ther of these was significant. The corresponding eta correlations were 
.64 and —.62, respectively. Both of the latter were significant. 

If the eta correlations are to be beheved, the redundancy 
variables were among the best used in the present study. Although the 
F test of linearity failed to demonstrate any curvilinearity, two other 
sources of evidence strongly suggest that curvilinearity did exist. First, 
the distributions of the passage redundancy scores were markedly 
skewed. The mean of LET RED4 for the total set of passages was .34 
while the standard deviation was .30. The mean and standard devia- 
tions of LET REDu were .46 and .26, respectively. In both cases 
skewing was indicated. Second, a visual inspection of the scatter plots 
of the correlations gave a distinct impression that the relationships 
were curvilinear. 

Independent clause frequency Independent clause frequency ap- 
peared to have httle, if any, usefulness as a predictor of comprehen- 
sion difficulty. At the independent clause level of analysis the product 
moment correlation was not significant. The eta, which was .I2, is 
probably not large enough to be of much use as a predictor of difficulty. 
At the passage level a similarly low correlation was observed. Although 
visual inspections of the scatter plots gave the impression that both of 
these relationships may have been curvilinear, they also confirmed 
that httle correlation was present at the start. 

F orm class and part of speech ratios The part of speech and form 
class ratios ranged from excellent to poor when judged in terms of the 
size of their correlations with passage difficulty. Table I7 shows the 
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correlations between each of the form class ratios and passage diffi- 
culty. Of the 25 correlations shown in this table, over half were sig- 
nificant at the .05 level. However, one must be cautious about inter- 
preting the sizes of these correlations. Visual inspections of the scat- 

Table I7 Correlations between passage difficulty and each 
of the part of speech ratios 



Denominator 



Numerator of the ratio 



of the ratio 


n 


pn 


V 


adv 


prep 


conj 


adj 


inti 


n 




23 


20 


21 


-46 


-34 


-36 


-61 


pn 


-36 


— 


-36 


-13 


-44 


-52 


-46 


62 


V 


-28 


31 


— 


12 


-46 


-57 


-48 


61 


adv 


-31 


-01 


-20 


— 


-39 


-51 


-44 


63 


prep 


54 


47 


51 


44 


— 


-15 


21 


55 


conj 


56 


81 


73 


66 


22 


— 


30 


61 


adj 


37 


39 


43 


47 


-11 


-25 


— 


58 


intj 


53 


54 


53 


55 


44 


52 


48 


— 


all other 


07 


39 


42 


33 


-42 


-44 


-43 


60 



* When r > .44, p < .05. 



ter plots again gave evidence that many of these regressions may 
have been curvilinear. 

Table 18 shows the correlations between the part of speech 
ratios and passage difficulty. Over half of these correlations were sig- 

Table 18 Correlations between passage 
difficulty and each of form class ratios 



Denominatori<‘ Numerator of the ratio 



of the ratio 


n 


V 


adj 


adv 


itr 


n 




13* 


-61 


10 


-66 


V 


-19 


— 


-57 


07 


-56 


adj 


55 


58 


— 


46 


37 


adv 


-14 


-12 


-34 


— 


-22 


str 


67 


67 


-39 


37 


— 


all other 


49 


49 


-56 


23 


-69 



<* Conventional part of speech names were 
given as an aid in interpretation of this table, 
but the correspondence of the form class to 
parts of speech is approximate. The abbrevia- 
tions used are read: form class 1-n-noun, form, 
class 2-v-verb, form class 3-adj-adjective, form 
class 4-adv-adverb, and Structural words-Str. 

* Where r > .43, p < .05. 



nificant. One variable, the ratio of pronouns to conjunctions exhibited 
one of the highest correlations yet observed for a single variable in a 
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readability study. Nearly all of the ratios containing interjections in 
either the numerator or the denominator proved to be very good nre- 
dictors of difficulty. S 

The contrasts between the correlations in Tables I7 and 18 
should also be noted. Some of the form classes contain nearly the same 
sets of words as some of the part of speech categories. For example, 
the noun category contained all of the words in form class 1 but 
form class 1 also contained a few words of other typos. This small 
difference resulted in rather dramatic differences in the sizes of the 
correlations that the variables exhibited. This is most clearly seen by 
comparing the sizes of the correlations involving the ratio of nouns to 
all words and form class 1 to all words. Similar contrasts can be made 
between form class 2 and the verb, form class 3 and the adjective, 
and foiin class 4 and the adverb. These contrasts suggest that sub- 
categories of words can be found which constitute still more powerful 
predictors of difficulty. 

Independent clause as a grammatical unit It appears that meas- 
ures of length based upon the independent clause yield higher corre- 
lations with passage difficulty than those based upon the sentence. 
The length of these two types of grammatical unit was measured by 
counting the letters, syllables, or words within them. Table I9 con- 
trasts the difficulty correlations of these measures. 

T able 1 g Measures of length based on 
the independent clause and sentence 
contrasted with respect to the sizes of 



their correlations 
culty 


with 


passage 


diffi- 


Unit 


Unit of measure 


measured 


Letter 


Syllable 


Word 


Independent Clause 
Sentence 


-.81 

-.67 


-.80 

-.70 


-.77 

-.58 



Unfortunately, there is presently no way to test the signifi- 
cance of the differences between these correlations, since these corre- 
lations are, themselves, probably highly correlated. But while it is 
impossible to test the significance of these differences, it seems' highly 
improbable that differences of this size could have occurred by chance 
even when only twenty passages are used. 



L 

o 

ERIC 



j 



Readability: A new approach bormuth 



II9 



This result may not be as simple as it at first appears; there is 
some evidence that sentence length and independent clause length 
may actually measure different aspects of style. The measures of sen- 
tence length tended to have higher correlations with the mean word 
depth variable than the independent clause length measures. When 
sentence length was measured in letters it was .81; measured in syl- 
lables, .83; and measured in words, .86. The corresponding correla- 
tions with independent clause length were .48, .5I, and .59, respec- 
tively. Consequently, independent clause and sentence length measures 
may actually measure different aspects of prose style. 

Units for measuring length The lengths of words, independent 
clauses, and sentences were measured using as units of measure let- 
ters, syllables, and words. The purpose was to find out which units 
of measure resulted in variables with the highest correlations with 
passage difficulty. Table 20 shows the resulting correlations. 

Table 20 Letters, syllables, and words as 
alternative units for measuring length, con- 
trasted with respect to correlations with 
comprehension difficulty 



Language unit measured 



Unit of 
measure 


Independent 

Word clause Sentence 




Word level of analysis 


Letter 

Syllable 


-.50 

-.40 




Independent clause 
level of analysis 


Letter 

Syllable 

Word 


-.50 -.43 

-.45 -.42 

-.45 




Sentence level of analysis 


Letter 

Syllable 

Word 


-.42 -.46 

-.35 -.46 

-.39 




Passage level of analysis 


Letter 

Syllable 

Word 


-.68 -.81 -.67 

-.80 -.80 -.70 

-.77 -.77 -.58 
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If one may judge from direct comparisons of the correlations 
in Table 20, it appears that counts of the numbers of letters provide 
the best measures of word length. Each of these correlations is from 
.05 to .lo higher than those obtained when syllables provided the unit 
of measure, and this occurred regardless of the level of analysis upon 
which the measures were used. When measuring the lengths of inde- 
pendent clauses, it appears to make little, if any, difference what unit 
of measure is used. On the other hand, it seems that letters and syl- 
lables may provide the best units for measuring sentence lengths. 

Words in prepositional phrases Table 2I contrasts variables ob- 
tained by counting prepositional phrases with variables obtained by 
counting words in prepositional phrases. The contrasts show that some- 
what higher correlations were obtained at the independent clause and 
sentence levels of analysis by using a count of the number of preposi- 
tional phrases. At the passage level it seemed to make little difference 
which method was used to derive the variable. 



Table 2I Number of words in prepositional phrases 
and number of prepositional phrases contrasted with 
respect to correlations with comprehension difficulty 



Unit 

meatured 


CountM based on— 

Words in 

prepositional Prepositional 

phrases phrases 




Independent clause 
level of analysis 


Independent Clause 


-.21 


-.32 




Sentence level of analysis 


Independent Clause 


-.19 


-.37 




Passage level of analysis 


Per Word 


-.46 


-.41 


Per Independent Clause 


-.70 


-.72 


Per Sentence 


-.72 


-.75 



Net measures The net measure is logically neither a measure of 
the lengths of words nor a measure of the lengths of sentences, but a 
composite of both. A similar statement may be made about the rela- 
tionships among measures of net, mean, and total word depths. Table 
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22 contrasts these three sets of measures at the independent clause 
and sentence levels of analysis. 



Table 22 Net, total, and mean meas- 
ures contrasted with respect to cor- 
relations with comprehension diffi- 
culty 



Unit of 
measure 




Measure 




Net 


Total 


Mean 




Independent clause 




level of analysis 


SyUable 


-.48 


-.42 


-.45 


Word Depth 


-.29 


-.33 


-.20 




Sentence level of analysis 


SyUable 


-.51 


-.46 


-.51 


Word Depth 


-.29 


-.34 


-.15 



- it the independent clause level net number of syllables seems 
to yield a slightly higher correlation with difficulty than either of the 
other two measures. Net depth, however, yielded a somev/hat higher 
correlation than the correlation obtained from total depth. Almost ex- 
actly the same relationships were obtained at the sentence level of 
analysis. It would appetir from this analysis that the net measures are 
fairly useful predictors of difficulty at the independent clause and sen- 
tence levels of analysis. 

Comparisons with previous studies A number of the best variables 
devised by earUer investigators were included to provide a frame of 
reference within which to judge the value of the new variables. In this 
case the value of a variable is judged by the size of its correlation with 
passage difficulty. In the absence of such variables it would have been 
uncertain whether the higher correlations exhibited by the new varia- 
bles were due to some superiority of the cloze technique used in this 
study or to the quaUty of the new variable itself. 

The correlations between passage difficulty and each of the 
previously studied variables are shown in Table 23. Table 23 compares 
the correlations found by earlier investigators with those found 
in the present study. These correlations were almost identical in 
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Table 23 Correlations between passage ditiiculty and lin- 
guistic variables used in both present and earlier studies 



Variable 


Earlier Study 




Present 

Correla- 

tion 


Author 


Correla- 

tion 


SYL/WOR 


Flesch (I950) 


.69 


.63 


syl/sen 


Gray and Leary (1935) 


.47 


.70 


WOR FRE/WOR 


Lorge (I948) 


.51 


.54 


PREP PHR/WOR 


Lorge (I948) 


.43 


.46 


DL 769 


Lorge (I948) 


.61 


.68 


DL 3000 


Dale and Chall (I948) 


.68 


.64 


WOR/SEN 


Flesch (I948) 


.52 


.58 



Note: Signs of correlations are ignored in this table. 



The most notable exception was in the case of syllables per 
sentence where the two correlations differed markedly. This difference 
was probably due to the fact that Gray and Leary used very short 
comprehension tests which may have had low reliabilities. 

Discussion The close agreement in the sizes of the correlations 
found in this and earlier studies provides one basis for judging the 
values of the new variables. However, one must exercise caution in 
doing so, since these variables have been compared only with respect 
to the size of their correlations with passage difficulty. Of at least equal 
importance are a variable’s correlations with other linguistic variables. 
But this type of analysis is best carried out through multivariate ana- 
lytic techniques and is beyond the scope of the present study. Hence, 
the present study must confine itself only to general statements about 
the magnitudes of the correlations. 

Mean word depth evidently measures a form of sentence 
complexity which is somewhat independent of the lengths of inde- 
pendent clauses and, to a lesser extent, of the lengths of sentences, 
evidenced by the fact that it entered the multiple regression equation 
with passage difficulty even after a measure of independent clause 
length had been entered into the equation. Further, it had a high cor- 
relation with comprehension difficulty. These facts are of considerable 
theoretical and practical importance. From the theoretical standpoint, 
it would seem that the length of a grammatical unit can be measured 
separately from its complexity, and both length and complexity seem 
to contribute independently to the difficulty of written material. 

It is likely that the word depth measure can be refined con- 
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siderably. For example, the present method of analysis forbids making 
trinary syntactic cuts. Many linguists would argue strongly for mak- 
ing trinary cuts as well. Also, the psychological assumptions under- 
lying the model are stated from the point of view of the difficulty for 
the writer or speaker. If stated from the point of view of the reader or 
listener, the prediction power of this variable might be further in- 
creased. 

The letter redundancy measures seem to be good predictors 
of comprehension difficulty. While their correlations with difficulty 
were not uniformly high, this v/as judged to be due to the facts that the 
relationships were curvilinear and that only short passages were used. 

It seems likely that several useful modifications can be made 
which may improve this measure. First, it was noted that redundancy 
calculated across four letters had a negative sign when correlated with 
difficulty, while redundancy calculated across eleven letters had a 
positive sign. This might be interpreted as showing that redundancy 
calculated for eleven letters was confounded and its correlation thereby 
reduced. If this were true, and if the effects of the first four letters 
were subtracted, the difficulty correlations of this variable might in- 
crease. It should be pointed out that this mathematical model could 
just as easily be applied to parts of speech to form a measure of gram- 
matical complexity. 

Initially the independent clause was thought of as being a 
more appropriate Uxiit of analysis than the sentence for measuring 
the lengths of grammatical units; and, indeed, the correlations with 
difficulty that involved this unit were among the highest observed. 
What was not expected was that, when the length measures were ap- 
plied to independent clauses, they seemed to be measuring something 
different from what was being measured when they were applied to 
sentences. Because of the relatively close correlation between the sen- 
tence length and the mean word depth measures, it was assumed that 
sentence length was, to some extent, a measure of grammatical com- 
plexity, while similar measures applied to independent clauses meas- 
ured length only. 

Other findings were of special interest. The letter appeared to 
be one of the most useful measures of the length of a unit. This is 
not really very surprising, since the letter offers a more discriminating 
metric than either syllables or words. The net measures resulted in 
variables having fairly high correlations with difficulty at the inde- 
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pendent clause and sentence levels. Counting the words in preposi- 
tional phrases may have its greatest value at the independent clause 
level of analysis, but, even here, the results were not substantially 
superior to those obtained by simply counting the prepositional phrases. 

Summary and conclusions 

Improvement in the accuracy of predicting and controlling 
the readability of printed language is vitally needed. Major research in 
readability almost ceased after l94^- Dormancy occurred because 1 ] 
there were no valid methods for measuring the comprehension diffi- 
culty of written language, and 2] there was r.o organized body of 
basic research and theory upon which readability research could draw. 
The development of the cloze test solved the problem of validly meas- 
uring difficulty while it simultaneously provided additional power and 
flexibility in making those measurements. The organization of psycho- 
linguistics as a discipline provided readability research with a body of 
basic research and theory, and linked readability research more closely 
to research and theory in psychology, linguistics, and literary style. 

While these developments opened vast new areas for advanc- 
ing knowledge of readability, they also revealed certain basic problems 
which must be solved before sound readability formulas can be devel- 
oped. The present study investigated what were considered to be most 
basic questions. These questions and the related results and conclu- 
sions from this investigation are summarized as follows. 

Are linguistic variables linearly related to comprehension dif- 
ficulty? If they are not, non-linear regression techniques must replace 
the linear equations used in present formulas. An F test of linearity 
was performed to determine if the regression between each linguistic 
variable and comprehension difficulty was linear. At the word level of 
analysis, all regressions departed significantly from linearity where a 
significant correlation existed to begin with. The shapes of the curves 
obtained suggested that readability formulas designed to predict diffi- 
culties of individual words must utilize quadratic equations. 

At the independent clause level of analysis, all but the regres- 
sion involving words on the Dale List of 3,000 words were linear. 
However, a note of caution must be injected in interpreting these re- 
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suits. First, several of the F ratios approached a significant level. An 
inspection of these scatter plots showed that, if curvilinearity existed, 
its effects were most pronounced at the extreme ends of the distribu- 
tions. Unfortunately, the F tests of linearity is least powerful in detect- 
ing curvilinearity at the extremes because so few cases occur there. 
Hence the failure to detect curvature may have been due to the limita- 
tions of present statistical techniques. Second, several variables were 
averages of one sort or another. It follows almost automatically from 
the central limits theorem that averages are normally distributed. 
Since two normally distributed variables almost invariably exhibit a 
linear relationship, the results of the F tests were more or less pre- 
dictable. A good illustration was that syllables per word exhibited a 
curvilinear relationship at the word level of analysis but a linear rela- 
tionship at the passage level. Consequently, it must be concluded that 
this problem must receive further investigatior. at the independent 
clause level of analysis to see if curvilinearity actuaUy does exist at 
the extremes and if transforming the variables before averaging or 
summing them increases the correlations. 

At the passage level of analysis, the results were similar ex- 
cept that no F ratio reached significance at the .05 level. That is, in- 
spections of the scatter plots seemed to reveal a tendency for several 
of the variables to exhibit curvilinearity at the extremes. This was 
especially evident in the scatter plots of the Dale List and letter re- 
dundancy variables. Since only 20 passages were used, the F test had 
even less power at this level of analysis. Further investigation of the 
problem was also needed at the passage level of analysis. 

Do linguistic variables more strongly influence difficulty for 
subjects at one level of reading ability than for subjects at other levels? 
If they do, different readability formulas must be developed for use 
with subjects at each level of reading ability. A five by three analysis 
of variance design was used to determine if linguistic variables influ- 
ence difficulty more strongly for subjects at one level of reading ability 
than for subjects at other levels. The subjects were stratified into five 
levels using their scores on the reading achievement test, and the 
comprehension difficulty of each word, independent clause, and pas- 
sage was calculated separately for the subjects at each ability level. 
The language units were then stratified into three levels using theii’ 
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values on a language variable, and the analysis of variance was cal- 
culated. The comprehension difficulties of the language units served 
as the dependent variable. A significant interaction between linguistic 
and ability levels in this analysis indicates that a variable is a better 
pedictor of difficulty for subjects at one level of ability than for sub- 
jects at other levels. 

The results show quite clearly that a linguistic variable can 
usually be expected to predict difficulty equally well for all subjects 
regardless of their reading levels. This generalization must, of course, 
be restricted to subjects whose reading achievement scores fall in the 
range represented in this study, second through twelfth grade in read- 
ing achievement grade placement score terms. But when cell means 
were plotted, no trend was evident which would suggest that this gen- 
eralization might not hold equally well for subjects of greater reading 
ability. Only one exception to this generalization, the independent 
clause frequency variable, was found. However, this was of little inter- 
est because of its low correlation with comprehension difficulty. Be- 
cause of ceiling effects, this problem could not be studied at the word 
level of analysis. However, this is not a serious limitation since all the 
variables occurring at the word level of analysis also occurred at the 
other levels of analysis where they did receive adequate investigation. 

Thus, it was concluded that a single readability formula can 
be used to predict difficulty for subjects at almost any level of reading 
ability. This greatly simplifies the task of developing readability 
formulas. Further, it shows that readability analysis should be applied 
to materials used with students at higher levels of education than has 
customarily been the case. The amount that even the most able stu- 
dents learn is accurately predictable from the character of the language 
in their instructional materials. 

Can useful readability predictions be made for language units 
as small as individual words, independent clauses, and sentences! If 
they can, formulas for making these predictions would be extremely 
useful for locating difficult parts of texts, c.nd predicting the compre- 
hension difficulties of test items, book iitdexes, titles, and headings. 
The objective was to determine if readability formulas designed to 
predict the difficulties of language units as small as individual words, 
independent clauses, and sentences have enough validity to wari'ant 



L 







J 



Readability; A new approach 



BORMUTH 



I27 



their further development. A multiple correlation was found between 
the linguistic variables and comprehension difficulty at each level of 
analysis. Had the multiple correlations been low and had the regres- 
sions evidenced curvilinearity, non-linear procedures would have been 
used. The multiple correlation was .5I «it the word level of analysis; .67 
at the independent clause level; and .68 at the sentence level. While 
an inspection of the scatter plots showed curvilinearity present, the 
multiple correlations were high enough to demonstrate the point with- 
out further treatment of the data. 

Readability formulas found useful in the past had validities 
ranging from .5 to .7. By these standards, the formulas represented 
by the regression equations found in the present study must be judged 
useful. Further, the formulas for predicting the difficulties of small 
language units can be greatly improved. The most obvious way to 
improve them is to deal with the problem of curvilinearity. At the 
word level this will produce a large and easily obtained increase in 
validity. At the independent clause and sentence levels, correcting for 
curvilincc rity must necessarily be less direct and a large gain in validity 
is less certain to result. The most important gains in validity will be 
obtained by developing more valid measures of the attributes of prose 
style. While this is the most difficult approach, it is the most impor- 
tant, for it will increase our understanding of the psychology of lan- 
guage processes and make important contributions to school curricula 
in both language composition and comprehension. 

Can the validities of readability formulas based entirely upon 
linguistic variables be improved? If they cannot, research efforts should 
be diverted to investigate variables which promise to yield gains. 
Historically, readability prediction formulas utilized only linguistic 
variables, variables based on objectively measurable features of lan- 
guage, and more or less ignored features that cannot be described ob- 
jectively. Can the validity of readability formulas based exclusively on 
linguistic variables be improved beyond the point already reached? The 
present study with its improved research techniques and large variety 
of linguistic variables offered an excellent opportunity to study this 
question. Two multiple regression equations were calculated using 
the variables at the passage level of analysis. The data were analyzed 
in two parts because of the large number of variables involved. One 
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multiple correlation was .93; the other, .81. None of the variables used 
in earlier studies was sufficiently powerful to enter the equations. 

It was concluded that much improvement is possible in the 
validities of readability formulas. The formulas calculated in the pres- 
ent study accounted for 30 to 60 per cent more of the comprehension 
difficulty variance than was possible with earlier formulas. The in- 
crease was due almost entirely to the greater validity of the new 
linguistic variables. Since these new variables only exemplify (and 
probably do so only crudely) the range of new variables it is possible 
to develop, it is not unreasonable to expect carefully constructed read- 
ability formulas in the future to exhibit validity correlations closely 
approaching l.oo. 

While this conclusion may at first seem too strong, upon fur- 
ther analysis it is quite reasonable. Readability formulas have -predic- 
tion validity only. The variables in them cannot be interpreted as 
causing the difficulty of text material. Difficulty is undoubtedly caused 
by the language itself, but these linguistic features constitute only a 
part of the causal v iriables. Semantic, organizational, and content 
variables undoubted y constitute other causes of difficulty. However, 
linguistic variables can index these other types of variables. For exam- 
ple, discussions of highly complex subject matters are usually accom- 
panied by many conjunctions, while discussions of interpersonal trivia 
are usually accompanied by the use of personal pronouns. Hence, the 
linguistic variables reflect the content variables in ordinary writing. 
Consequently, it is not unreasonable to expect readability formulas 
based exclusively on linguistic variables to predict nearly all of the 
difficulty variance among texts. But it would be most unreasonable to 
assert that the linguistic variables in those same formulas caused 
nearly all of the difficulty variance. A clever writer could quickly dem- 
onstrate that such a statement is nonsense. 

Another point that merits discussion deals with the very large 
correlations which will be routinely observed in modern readability re- 
search. Psychologists have become accustomed to rechecking their cal- 
culations for errors when they obtain correlations in excess of .8. This 
is largely because their measurements usually contain a great deal of 
error. In readability research, error is much smaller. The linguistic 
variables are based upon exact counts and contain little or no error. 
Difficulty measurements are based upon test means which involve only 
the errors of the means which are much smaller, usuaUy, than the 
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errors of individual test scores. Consequently, readability correlations 
of .98 and .99 are not impossible and will probably be looked upon as 
commonplace in the future. As long as readability correlations are not 
interpreted as implying causation, these high correlations need not 
cause intellectual indigestion for anyone. 

Does the use of various new types of linguistic variables ini' 
prove the accuracy of readability prediction? If the accuracy of pre- 
diction is improved, the underlying rationale of new linguistic variables 
should be fully exploited to explore and refine them. A large number 
of previously uninvestigated linguistic variables were included in the 
present study. Some were representative of fundamentally new ap- 
proaches to the measurement of prose style. Others were simply refine- 
ments of well known variables. The purpose was to determine if varia- 
bles of these types warranted further development. The methods used 
were appropriate for exploratory purposes only; the value of a variable 
was judged almost solely by the size of its correlation with comprehen- 
sion difficulty. When comparisons were made between variables, tests 
of significance were not attempted because the nature of the data made 
tests of significance impossible. 

Because of the large number of variables involved, only the 
most important results can be discussed here. Perhaps of greatest im- 
mediate importance was the fact that the length and complexity of a 
sentence can be measured separately; and, though length and com- 
plexity are correlated, each has an independent correlation with diffi- 
culty. The extent to which their mutual correlation is necessary is an 
important problem for linguistic research. Another important finding 
was that the part of speech variables exhibited very high correlations 
with difficulty. For example, the ratio between the numbers of pro- 
nouns and conjunctions in a passage exhibited a correlation of .81 
with difficulty. The third major result was that many of the well 
known variables were markedly improved by making relatively minor 
refinements in the way they are derived. 

If any general conclusion is warranted by the results from this 
part of the study, it is that the major advances in readability predic- 
tion will result from the development of more sophisticated variables 
with which to measure the attributes of prose style. Whatever the best 
possible linguistic variables may be, it is certain that only a beginning 
has been made in their discovery. 
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Implications 

Readability research can again make rapid strides toward the 
achievement of its two main goals, readability prediction and readabil- 
ity control, and this research must take new directions. Without ques- 
tion the most important advances should come through the develop- 
ment of better linguistic variables developed through the study of 
psycholinguistics, linguistics, and literary style. 

The wholesale introduction of new variables can have only 
limited value unless some method is used to integrate each new lin- 
guistic variable into a general body of readability theory and unless 
methods are used whereby large numbers of linguistic variables can 
be simultaneously evaluated. Factor analysis techniques seem to offer 
an excellent method of solving both problems. Theory in the area of 
literary style is poorly developed, is stated in vague subjective terms 
and contains few statements of how the variables of prose style relate 
either to each other or to responses in a reader. Factor analysis of 
linguistic variables can reveal the basic dimensions in which prose 
style varies among authors, thus providing the theoretical framework 
within which the properties of new linguistic variables can be studied. 
The factors or dimensions of prose style can themselves be studied to 
determine how each influences responses in a reader. 

This analytic strategy will greatly facilitate experimental read- 
ability research. The ultimate goal of readability research is to gain the 
ability to control the difficulty of language. This can only be achieved 
through the use of experimental methods which make it possible to 
determine if a linguistic variable is causally related to difficulty. At 
present, it is absurd to attempt to apply experimental techniques to the 
study of readability because of the great number of variables that must 
be controlled. Over I50 linguistic variables have been shown to cor- 
relate with difficulty. If these variables could be held constant, only a 
few (’ays’ work would be required to devise another I50 variables that 
would also have to be controlled. Fortunately, many of these variables 
are closely related to each other. Through factor analytic techniques 
redundancies can be removed and the entire set of variables reduced 
to a few basic style factors. This small number of factors can then be 
easily managed in experimental designs. 

The introduction of cloze tests into readability research solved 
the most urgent and important psychological measurement problem in 
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readability. There are still no adequate instruments for measuring the 
interest and aesthetic responses passages elicit in subjects. As a result 
these important aspects of readability ha\^e not been dealt with by 
careful researchers. One possibility that should be explored is to de- 
termine whether the principles of the semantic differential can be 
adapted to construct instruments for these purposes. 
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